This week's issue has posts from Scribd and Slack on Apache Airflow, using Envoy with Apache Kafka, the open sourcing of LinkedIn's DataHub, Elasticsearch in production, and Apache Flink's new SQL DDL support. Also a post on the data infrastructure behind the "Spotify Wrapped Campaign" and an article with advice on running a data team.

Slack wrote about their experiences upgrading Apache Airflow from version 1.8 (which they had been running for two years) to version 1.10. The post describes the upgrade strategies that they considered, the steps they took (many around schema changes and backing up the metadata database), how they tested the upgrade, and some issues they found after the upgrade.

https://slack.engineering/reliably-upgrading-apache-airflow-at-slacks-scale-2a31f3d03a06

Spotify writes about their large scale analysis of a decade of playback data to power their "Spotify Wrapped Campaign" at the end of 2019. They performed a number of intermediate jobs, which allowed them to more quickly iterate and verify the quality of outputs. They talk about some of the changes they made since 2018's campaign, including changing the way that they store data in order to avoid large amounts of shuffling (and thus higher processing costs).

https://labs.spotify.com/2020/02/18/wrapping-up-the-decade-a-data-story/

Scribd writes about their journey from a home grown workflow engine to Apache Airflow. Their main DAG has over 1400 tasks (there's a fun visualization in the post), so it's a big undertaking to make the move. The post describes the main motivators, and some of the high-level changes they've made to move to Airflow.

https://tech.scribd.com/blog/2020/modernizing-an-old-data-pipeline.html

A mix of technical and managerial advice, this post shares lessons learned from running the data team at Gitlab for a year. Technical topics include how to choose the right tools (including strategically buying some products) and investing in process/tools for onboarding. And if you're on the manager side, there's a bunch of advice about how big your team should be, how to get executive buy in, and more.

https://about.gitlab.com/blog/2020/02/10/lessons-learned-as-data-team-manager/

An introduction to using Envoy 1.13 as a reverse proxy for Apache Kafka traffic. The post describes how to configure an Envoy filter to gather metrics on requests/responses for client traffic-only or all traffic (including requests for inter-broker replication).

https://medium.com/@adam.kotwasinski/deploying-envoy-and-kafka-8aa7513ec0a0

LinkedIn has open sourced DataHub, their tool for metadata management of data platforms. While the technical details of DataHub were covered in previous posts, this article describes how LinkedIn plans to maintain the code both internally and as an open source project, and also how the features of the two versions differ.

https://engineering.linkedin.com/blog/2020/open-sourcing-datahub--linkedins-metadata-search-and-discovery-p

Apache Flink 1.10 adds new SQL DDL syntax for configuring data sources and sinks. The post has some examples for defining new tables (e.g. Kafka and Elasticsearch) and details on Flink's catalog system.

https://flink.apache.org/news/2020/02/20/ddl.html

The morning paper has coverage of a paper on Microsoft's Raven, which embeds ML runtimes into SQL Server. The idea is to keep models as data in the database so that you can take advantage of features like transactions and improve performance. Pretty interesting details, including how they've released as part of public preview in Azure SQL Database Edge.

https://blog.acolyer.org/2020/02/21/extending-relational-query-processing/

An in depth look at the architecture of Elasticsearch towards the goal of planning and monitoring a production deployment.

https://facinating.tech/2020/02/22/in-depth-guide-to-running-elasticsearch-in-production/

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.

Lots of great stuff in this week's issue, including the release of MR3, a couple of posts on schema migration (including one on CI/CD for Redshift), how Bitbucket has scaled their databases, and more.

HERE Mobility has written about their CI/CD pipeline for Amazon Redshift. They've built a tool to apply database schema changes, validate the structure of the database (e.g. find broken views), verify redshift SQL syntax, and automatically deploy to Redshift. Lots of good tips in the post about automating validations and using version control for SQL scripts.

https://medium.com/big-data-engineering/redshift-cicd-how-we-did-it-and-why-you-should-do-it-to-e46ecf734eab


A look at the trade-offs between various types of data driven architectures, like event sourcing, change data capture, command query response segregation, and the outbox pattern. The post dives deep into the architectures, including clarifying the difference between domain events and change events. There are some useful diagrams that help to better understand how the various pieces fit together in each architecture pattern.

https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/


The MR3 framework, which offers an alternative execution model to YARN and MapReduce for Hive/Spark/Hadoop workloads, has released version 1.0. This release includes a number of improvements for a deployment in the cloud, including improvements for S3 and Amazon EKS.

https://groups.google.com/forum/#!msg/hive-mr3/3VwpqBnZfT4/9emGzbZ9BQAJ

This post describes how to use the open source osquery project to collect data and send it to Apache Kafka as part of a security information and event management platform. The post describes the basics of osquery and how to build a custom extension, written in python, for producing data to Apache Kafka.

https://www.confluent.io/blog/siem-with-osquery-log-aggregation-and-confluent/

Github writes about how they've automated the process of applying schema migrations at scale (meaning number of tables, number of developers, and size of server fleet). Using Github Actions and the tools skeema and skeefree, they create "safe" (i.e. no direct DROP TABLE and also rewriting ALTER TABLEs to be efficient) schema migrations by comparing the current table definition in source code to what's defined in the database. The post describes both these tools (which are a bit specialized for MySQL) and the workflow (which includes chatops and github actions).

https://github.blog/2020-02-14-automating-mysql-schema-migrations-with-github-actions-and-more/

A look at performance with Ozone, the blob storage layer for HDFS. In comparison to tests run directly against HDFS, when using the TPC-DS benchmark most of the queries run faster with Ozone as the storage layer.

https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/

Bitbucket writes about how they've scaled their databases by moving reads to replicas. To ensure that queries go to read replicas containing up to date data, they keep track of a per-user log sequence number from Postgres (stored in Redis). They also share how their changes have improved the performance profile of their databases.

https://bitbucket.org/blog/scaling-bitbuckets-database

An intro to five data engineering projects that are worth checking out, if you're not already using them: dbt (managing SQL code), Prefect (a new workflow engine), Dask (distributed computing for Python), DVC (the "data version control"), and Great Expectations (for testing data). In addition to those, the post calls out a few other projects worth investigating.

https://medium.com/@squarecog/five-interesting-data-engineering-projects-48ffb9c9c501

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.

Still catching up a bit on all the articles from my hiatus, and I pulled in posts from that time on Netflix's DBLog and the `xsv` CLI tool. In general, there’s lots of breadth in this week's issue—running Presto on Kubernetes, improvements to consumer flow control in Apache Kafka 2.4.0, building a queryable dataset with ksqlDB, Microsoft's automated analytics service for large scale deployments, and a look at migrating to CockroachDB.

Netflix writes about DBLog, their change data capture framework that integrates with MySQL, PostgreSQL and more. Probably its most novel feature is a mechanism for creating a dump of the entire database by chunking up the keyspace and processing data without a lock. DBLog also has a solution for high availability, and Netflix plans to open source it later this year.

https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b

A look at how to use the `xsv` command line tool to join data from multiple csv files. There are a few other tools recommended in the comments, too.

https://www.johndcook.com/blog/2019/12/31/sql-join-csv-files/

An overview of running Presto, its dependencies (like the Hive metastore and a relational database), and the redash query editor/visualizer on a Kubernetes cluster. The configuration, as well as code for some of the trickier bits like initializing/migrating the metastore database tables, is available on github.

https://medium.com/@joshua_robinson/presto-powered-s3-data-warehouse-on-kubernetes-aea89d2f40e8

Lightbend describes how the Alpakka framework relies on the Kafka Consumer's flow control (i.e. pause/resume) features, and the improvement in Apache Kafka 2.4.0 that drastically improves throughput and reduces bandwidth on brokers. At a high-level, the solution takes better advantage of data that is prefetched by the Kafka Consumer before `pause` is called.

https://www.lightbend.com/blog/alpakka-kafka-flow-control-optimizations

Presto added support for the SQL standard syntax for limiting query results, which is `FETCH FIRST n ROWS`. Vs the `LIMIT` syntax, this new one has some additional functionality for duplicates/ties.

https://prestosql.io/blog/2020/02/03/beyond-limit-presto-meets-offset-and-ties.html

An introduction to using ksqlDB to build a queryable dataset, including how load raw data into Apache Kafka, perform some transformations, and execute a SQL query to lookup individual results using an HTTP endpoint.

https://www.confluent.io/blog/build-materialized-cache-with-ksqldb/

This article describes what it's like to move from Postgres to CockroachDB. The main reason to switch was Cockroach's scalability, and there are some trade-offs (and new features!) that it comes with (e.g. no partial indexes or deferring of foreign key checks). Lots of good info if you're considering a similar migration.

https://www.openmymind.net/Migrating-To-CockroachDB/

A look at what the trade-offs are when it comes to sizing and density of disks for data nodes in a HDFS cluster. It seems like it's a good idea to choose more small disks, because this decreases how long it takes to detect bit rot and the time to recover from node failure/decommissioning.

https://blog.cloudera.com/disk-and-datanode-size-in-hdfs/

The Apache Flink blog has a post describing how to write unit tests for Flink operators. Stateless operators are pretty straightforward, and there are some examples for stateful and time operators.

https://flink.apache.org/news/2020/02/07/a-guide-for-unit-testing-in-apache-flink.html

Microsoft has published a paper on Gandalf, their tool for monitoring the status of deployments across Azure infrastructure. It correlates signals (logs, metrics, events) as the rollout of a new component version progresses and can stop a rollout based on what it observes. When it does, Gandalf describes which signals caused it to stop. The paper has lots of details and several case studies.

https://www.microsoft.com/en-us/research/publication/an-intelligent-end-to-end-analytics-service-for-safe-deployment-in-large-scale-cloud-infrastructure/

An overview of the architecture of Google Cloud Spanner, including details about the implementation of TrueTime, which is used to guarantee consistency across regions, and the lifecycle of reads and writes. There are lots of diagrams taken from public videos/talks on Spanner.

https://thedataguy.in/internals-of-google-cloud-spanner/

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.

Hey all! it's been a while—hopefully everyone had a nice new year. I took the last ~6 weeks off, so there's quite a bit to catch up on. This week's issue has sixteen of the best articles from that time covering topics like Apache Kafka producers, distributed storage engines for Prometheus, Presto+Pinto, and Jepsen analysis of etcd. Also, Yelp writes about their Kafka architecture, and Teads writes about optimizing Spark applications using User Defined Aggregate Functions. Lots to read up on, whether you're looking for some tips to apply to your own system, new tools to try out, or learning more about how systems work under the hood.

A look back at the themes that have emerged from the last decade in technology, including a number of distributed systems items like the return of SQL and streaming. The post also predicts big areas for the next few years, like future of PaaS and Kubernetes (and also areas with no connection to distributed systems like retail, journalism, and social media).

https://medium.com/@copyconstruct/a-decade-in-review-in-tech-1cde76c9b43c

A look at the important configurations to improve throughput of your Kafka Producers, as well as the key metrics to monitor on the broker related to efficient producing.

https://www.jesseyates.com/2020/01/01/high-performance-kafka-producers.html

This post covers several solutions for running large scale prometheus deployments. These include Thanos, Cortex, M3DB, and VictoriaMetrics. There are a mix of architectures—some push and some pull as well as various trade-offs for things like cold storage backends.

https://monitoring2.substack.com/p/big-prometheus

This post describes using SQLite to replace a networked database, which is quite an interesting idea. By sharing SQLite as a datastore across containers, they're able to improve latency by over 20x for a system that's processing 500k messages per second. Several details to dig into here, and some commentary on the larger impacts of a change like this (which pushes complexity out of a data plane and into the control plane).

https://medium.com/@rbranson/sharing-sqlite-databases-across-containers-is-surprisingly-brilliant-bacb8d753054

A good look at how the Java Virtual Machine manages heap space, and how the garbage collector interacts with various memory regions in order to reclaim space. The post describes the various types of garbage collection strategy, the types of events that trigger a garbage collection, and more.

https://sematext.com/blog/java-garbage-collection/

A thorough introduction to connecting to databases from Java, covering JDBC, Hibernate, the Java Persistence API, lightweight libraries for Java SQL, and more. The post has lots of code examples, advice for when certain libraries are appropriate, and more.

https://www.marcobehler.com/guides/java-databases-jdbc-hibernate-spring-data

Presto's 2019 year in review—covering new syntax (e.g. adding comments and fetching just the first N rows), query optimizations (e.g. improvements to the cost based optimizer and lazy materialization), new connectors (elasticsearch, google sheets), and much more. The post also looks at what's next for Presto in 2020.

https://prestosql.io/blog/2020/01/01/2019-summary.html

Uber writes about how they've integrated Pinot, their real-time analytics system, with Presto for SQL queries. The article describes the architecture, and how they improved the connectors performance with predicate/limit/aggregate/more pushdown, and how it performs in practice.

https://eng.uber.com/engineering-sql-support-on-apache-pinot/

A look at instrumenting apps written in C, Java, and Golang for analysis using eBPF. The post is part of a larger series on eBPF, and there's an introduction to the main concepts at the top of the article.

https://sematext.com/blog/ebpf-userland-apps/

Quarkus, the java application framework, has a new extension to implement the outbox pattern (quite interesting to read about if you've missed earlier articles!) for change data capture. The Debezium blog has an introduction that walks through how to get started with Quarkus and the Outbox Quarkus Extension for generating events.

https://debezium.io/blog/2020/01/22/outbox-quarkus-extension/

Yelp writes about their Kafka infrastructure, which has several components. The post describes two of them in detail, the Stream Discovery and Allocation service (which enforces schemas and defines a stream as either fire and forget or acked) and "Monk Leaf" which is a service that runs locally and proxies to Kafka (implementing either the acked or fire and forget semantics). These components provide a platform that make it easy for developers to deploy applications and get data into the Kafka data pipeline.

https://engineeringblog.yelp.com/2020/01/streams-and-monk-how-yelp-approaches-kafka-in-2020.html

The third post in a series on Spark Job Optimization myths (the previous two looked at executor memory and number of executors), this post looks at why adding more memory to the Spark driver doesn't always improve performance. It has some tips that can improve the driver performance instead—like avoiding globals and avoiding expensive calls to functions like `collect()`. If you're looking at optimizing your Spark usage, this series is worth digging into.

https://www.davidmcginnis.net/post/spark-job-optimization-myth-3-i-need-more-driver-memory

A post from Teads describes both how to speed up a Spark application using a Spark User-Defined Aggregate Functions (UDAF) as well as how to optimize Spark applications in general. Their article walks through how they sped up one of their applications from 28 mins to 9 mins. Several of the optimizations are informed by the Spark execution DAG, several of which they analyze in the post.

https://medium.com/teads-engineering/apache-spark-udaf-could-be-an-option-c2bc25298276

An overview of the how the Apache Kafka idempotent producer works, including pointers to relevant pieces of the code that generate ids, complete batches, and more. The post details how the producer has been extended since the original implementation to support multiple in flight requests.

https://www.waitingforcode.com/apache-kafka/apache-kafka-idempotent-producer/read

Jepsen has a post on etcd, which is the key-value store used by Kubernetes and other distributed systems. The article describes several tests of correctness, which verified strict serializable operations and correct delivery of watches. They found some issues with locks, which the etcd team is addressing (more details in a companion blog post). As always, it's great to read about verification of distributed systems—the post brings together practice and theory in a way that builds extra context on both.

https://jepsen.io/analyses/etcd-3.4.3

Events

Curated by Datadog ( http://www.datadog.com )

Quick editorial note: this week is the last issue in which I'll be including links to events.

California

Uber’s Big Data Platform: 100+ Petabytes with Minute Latency (Sunnyvale) - Tuesday, February 4

https://www.meetup.com/Introduction-to-Data-Science/events/267974206/

Stream Processing with Apache Kafka & Apache Samza (Sunnyvale) - Wednesday, February 5

https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/267283444/

Data Engineering Meetup (San Diego) - Thursday, February 6

https://www.meetup.com/Data-Engineering-San-Diego/events/268297626/

Colorado

CSOSUG: Kafka (Colorado Springs) - Monday, February 3

https://www.meetup.com/csopensource/events/267071302/

Missouri

Apache Airflow (Saint Louis) - Wednesday, February 5

https://www.meetup.com/St-Louis-Big-Data-IDEA/events/266391947/

BRAZIL

JVM & Kafka (Sao Paulo) - Thursday, February 6

https://www.meetup.com/SouJava/events/268154472/

SPAIN

Real-Life Architectures to Solve Real-Life Business Cases at AWS (Madrid) - Tuesday, February 4

https://www.meetup.com/Madrid-Data-Engineering/events/267986201/

Fast Track to Spark SQL & Structured Streaming (Barcelona) - Thursday, February 6

https://www.meetup.com/Spark-Barcelona/events/268116178/

FRANCE

Happy New Year! Paris Kafka Meetup (Paris) - Thursday, February 6

https://www.meetup.com/Paris-Apache-Kafka-Meetup/events/268164461/

SWITZERLAND

Apache Kafka Meetup at RBS (Zurich) - Thursday, February 6

https://www.meetup.com/Zurich-Apache-Kafka-Meetup-by-Confluent/events/267684714/

AUSTRALIA

Sydney Data Engineering Meetup (Sydney) - Thursday, February 6

https://www.meetup.com/Sydney-Data-Engineering-Meetup/events/266033376/

This week's issue includes articles on Pinterest’s time series database, deploying Elasticsearch on Kubernetes, data & ML engineering at Slack, Airbnb's job queuing system, and the architecture of the Cliqz search engine. There are also a slew of posts on other topics from optimizing Kafka consumer to JSON processing in Golang.

A look at the inner workings of the Kafka consumers, with some real world recommendations for deploying them when there's high latency in talking to the Kafka cluster and/or a large number of partitions. There are tips on important metrics to monitor, configurations, garbage collector settings, and changing the partition.class to improve unbalanced consumers.

https://www.jesseyates.com/2019/12/04/vertically-scaling-kafka-consumers.html

This article looks at why hash maps (unsorted) are popular for in-memory indexes whereas b-tree (sorted) are common in databases. It describes the trade-offs of the two approaches, and how those best fit in-memory/db use cases.

https://www.evanjones.ca/ordered-vs-unordered-indexes.html

This post describes a large scale conversion of data into JSON format in order to load it into BigQuery. To meet the naming requirements of BigQuery, they had to remap field names on every JSON document. Their tool, which is written in Golang, uses a producer/consumer job queue to parallelize processing and partition the data before writing it out. They processed data both from Kafka and S3, and the post talks a bit about how they optimized interaction with S3.

https://itnext.io/parsing-18-billion-lines-json-with-go-738be6ee5ed2

Pinterest writes about how they've extended their time series data store, Goku, to support querying of historical data. They tier data by compacting data through rebucketing and downsampling. For serving, they load data from S3 into RocksDB. The post goes into the details of the design of their RocksDB setup, cluster management functions, and the query processing framework.

https://medium.com/pinterest-engineering/gokul-extending-time-series-data-storage-to-serve-beyond-one-day-52264307364d

An overview of Kubernetes and how to deploy Elasticsearch on Kubernetes. This is a great introduction to many of the core concepts of Kubernetes (e.g. Deployment, Pod), including those that are important for running a stateful service (e.g. StatefulSet, PersistentVolumeClaim). It also shows how to configure accounts for ES using Kubernetes RBAC and use the helm package manager for deploying to Kubernetes.

https://sematext.com/blog/kubernetes-elasticsearch/

Available both in audio form and as a transcript, InfoQ has a podcast with Josh Wills that covers the evolution of data engineering and machine learning at Slack. The interview covers their data pipeline, which feeds into big data systems for BI/data warehousing and ML products. The interview also covers the kinds of products they build with machine learning and some thoughts on the future of observability for ML pipelines.

https://www.infoq.com/podcasts/slack-building-resilient-data-engineering/

Airbnb has open sourced Dynein, their job queuing system that they use for offloading tasks from the main request path and performing other asynchronous operations. It uses DynamoDB as a scheduler for future jobs and SQS for queuing—the post describes how this is built in a highly scalable way.

https://medium.com/airbnb-engineering/dynein-building-a-distributed-delayed-job-queueing-system-93ab10f05f99

When you're running a high throughput system in Java, issues with garbage collection are inevitable. This post provides details on how to enable GC logs, how to interpret the details from the Concurrent Mark Sweep and the G1GC collectors, and some tools for visualizing the output. This is one of the most comprehensive guides that I've seen on the topic.

https://sematext.com/blog/java-garbage-collection-logs/

Cliqz, makers of a web search engine, have a comprehensive post on their architecture. Their near real-time indexing system is built with Apache Kafka, Apache Cassandra, and RocksDB while their batch indexing system is built on MapReduce and Spark with Luigi for managing workflows. The post also describes how they manage Kubernetes clusters, use Helm/Helmfile for package management, and leverage Tilt and K9s for local development. They also share on how they optimize costs and describe their machine learning pipelines.

https://www.0x65.dev/blog/2019-12-14/the-architecture-of-a-large-scale-web-search-engine-circa-2019.html

Events

Curated by Datadog ( http://www.datadog.com )

California 

Building a Best-in-Class Data Lake on AWS and Azure (Santa Clara) - Tuesday, December 17

https://www.meetup.com/datariders/events/266951424/

North Carolina

Zero to Observability with Apache Kafka (Raleigh) - Tuesday, December 17

https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/266917829/

CANADA

Introduction to Kafka (Montreal) - Wednesday, December 18

https://www.meetup.com/montreal-jug/events/266729844/

BRAZIL

Data Engineering Meetup (Belo Horizonte) - Wednesday, December 18

https://www.meetup.com/engenharia-de-dados/events/267072117/

SERBIA

Event Deduplication in Kafka + Navigation in a 3D Environment with RL (Novi Sad) - Wednesday, December 18

https://www.meetup.com/Big-Data-Novi-Sad/events/267060354/

ISRAEL

Spark Advanced Topics (Tel Aviv-Yafo) - Thursday, December 19

https://www.meetup.com/Women-in-Big-Data-Israel/events/266728256/

RUSSIA

Real-Time Data: Streaming and Collecting Data in Real Time (Moscow) - Wednesday, December 18

https://www.meetup.com/Data-People/events/266992802/

INDIA

Designing ETL Pipelines with Structured Streaming and Delta Lake (Bengaluru) - Wednesday, December 18

https://www.meetup.com/Bangalore-Apache-Spark-Meetup/events/266970481/

Apache Kafka and Stream Processing Meetup @ Walmart (Bengaluru) - Sunday, December 22

https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/

SRI LANKA

Insights into HDInsights (Colombo) - Wednesday, December 18

https://www.meetup.com/sldatacommunity/events/267042058/

SOUTH KOREA

Flink Meetup (Seoul) - Tuesday, December 17

https://www.meetup.com/Seoul-Apache-Flink-Meetup/events/266824815/

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.