Kronos - A Generalized Data Orchestration Engine

Context

At Wayfair (they've got what you need!) from 2015-2020, I led teams that both built and leveraged Kronos, a generalized orchestration engine for data warehouses. The engine ended up having quite significant adoption, though it was too tightly coupled to Wayfair technology to be a viable open-source option. Post Wayfair, I realized that though I had a good sense of how Kronos fit into the broader landscape of data orchestration tools - of the time Oozie, Airflow, Luigi, SSIS, and more - and the tools of the future - DBT, etc - there were many who probably ran into it as a weird internal elephant. This post is intended to be a map of where it fit into the overall landscape of data tooling at the time, and how tools like DBT evolved past it.

As a reader, you fall into one of 3 audiences: you used Kronos in the past and feel like it was a fever dream; you somehow learned of it some other way and wonder what an HR company has to do with data processing at Wayfair; and those that work with Kronos today, if it still exists, for whom I am truly sorry. I don’t know what situation you currently have to deal with, but here’s how it got that way.

A note on the latter - I’ll use the present tense throughout this document, though Kronos may be long-gone.

What is it?

Kronos was a tool for orchestrating DAGs - Directed Acyclic Graphs - like most core ETL tools.

Kronos had 3 core principles that made it somewhat unique:

Automatic Dependency Resolution

Inclusion in a schedule - a DAG with a single ID - was always explicit, but the order of operations within a DAG was always implicit. Users never manually ordered dependencies globally. Concepts like "listeners" or "waitfors" used to poll in other tools (Airflow, etc) were not required for an asset that was managed by Kronos directly; these still existed as interfaces to external systems.

And these DAGs were big - our core daily processing graph spanned several thousand tasks, including massive dynamic fanout processing. Imagine an Airflow cluster, but all DAGs are merged.

Global SLA Based Prioritization

Tasks were prioritized globally, across all DAGs. Kronos would attempt to optimize landing times and resources with full knowledge of available resources and all work to be done.

Dynamic DAGs as a Core Feature

DAGs were mutable as they ran. It was extremely common for a DAG node to spawn additional tasks within the same DAG, which could lead to some very dynamic execution patterns.

And many that were not unique, but were important to success:

• Ran out of source control (SVN, then Git), with native support for reading from many repos that could be owned/managed by other teams
• Serialized tasks (the “compilation process”) into an internal database, which avoided expensive run-time evaluation and supported static analysis/resolution of immediate dependencies
• Robust templating support
• Data quality checks and notifications

Why was it?

These principles flowed from the problems Kronos was trying to solve. Kronos was created by a team trying to get rid of SSIS, with the following guiding principles:

Every task should be in source control, and a full software development life cycle should be possible. The barrier to entry should be extremely low - a user should not need to know Kronos-specific syntax for any common tasks. All kinds of jobs should be supported - from many flavors of SQL, to Python, to R, to arbitrary Docker containers. Explicit reuse of common data assets should be simple, easy, and popular.

And even more specifically, Kronos was built to process large SSAS cubes very fast. At the time of creation, Wayfair had multiple-terabyte SSAS cubes - Kronos would process these cubes to infer automatic dependencies and dynamically create per-partition XMLA processing jobs and optimistically schedule partition updates as soon as the underlying fact was available. The optimization of the SQL before the cubes was a nice side effect that later became the main goal.

Scalability

In my time, Kronos processed millions of tasks each day, in a mix of daily batch DAGs, microbatch DAGs every 15, 30, or 60 minutes, and occasional one-off tasks. These tasks were coordinated by an active/active set of schedulers (up to an arbitrary number) that prioritized a global queue with a sub-minute SLA, and a larger set of workers that were assigned specific queues to process and might further federate that work to remote systems like databases.

The prioritization algorithm used an initial eager scheduling heuristic based on the sum of downstream execution time within a DAG, supplemented by a slower asynchronous prioritization algorithm that optimized for landing times.

This applied to optimally structured SQL processing - where every query that created a table was a subnode that could be eagerly scheduled as soon as possible.

How this looked in practice

In one period, Kronos was responsible for orchestrating a global codependency graph across 5 distinct Vertica clusters (+ many other systems). Each day, Kronos would execute a dynamic graph that would build tables on some clusters, copy them to others, derive new assets, and potentially even copy back to the first clusters for more processing. This was in service of daily updates to a large set of dashboards, OLAP cubes, and other reporting.

A new data engineer - or analyst - creating a new table would fire up their IDE and write code to create a new table.

Once they had this working, they would edit this script to template in any variables they wanted - the most common by far being the ‘date’ the table was scheduled on, or a range - start_date to end_date - associated with the schedule, and commit this to a branch in source control.

At this point, the script could look like:

CREATE OR REPLACE my_fun_table AS SELECT My_id, My_value From my_table Join my_other_table on my_table.id = my_other_table.id WHERE My_table.date between ‘’ and ‘’

At this point, they could immediately execute the code in Kronos from the branch. Kronos would detect if it needed to recompile the script (such as if it was new), pull it from source control, parse out the dependencies, and run it. Kronos would know that this table, in a given schedule, would need to run after it had updated ‘my_table’ and ‘my_other_table’.

To get it scheduled, the user would typically have to merge it into their ‘main’ branch, which was one of a configurable set of schedulable branches. As soon as their pull request was approved, it would be pulled into any schedule assigned to that folder in Git and run automatically.

What worked well?

There is very little specific syntax required - the user is committing exactly what they could run in an IDE or DB tool, with some light variables that the IDE may support as well.

The user did not have to worry about the upstream tables, when they built, or what their schedule was.

What were the challenges?

The strict coupling to SVN/Git caused some challenges for the UX and for adoption by analysts - for many, it was the first time they encountered source control.

Our UI was... not great. Agentic CLIs would have been a huge help at the time.

SQL parsing is hard! Edge cases could cause challenges.

Initially, we coupled execution environments for Python code with the service code and its execution environment. We had to teach people not to call exit().

Security! We built a very trusting model to start; this definitely had both security + performance implications and had to be refined over time as we scaled to a large user base.

And much, much more…

Comparison to Other Tools

Airflow

We used Airflow extensively, especially for ML cases. Sometimes we would have composite pipelines that would span both platforms. Airflow was the default option for self-contained ML workflows, while Kronos was used more heavily for the larger warehouse graph.

Key differences:

DAG Structure. Kronos used a global DAG of dynamically assembled task nodes based on a common derived DAG ID, rather than having a DAG be explicitly assigned.

Dependencies. Kronos automatically constructed the dependency graph based on input/output annotations on the operator that related to a URI, rather than assigning dependencies between operators directly. These input/output annotations were inferred automatically for common patterns like SQL.

Resource Management: Kronos included a global resource management and optimization framework, based on an estimated processing time for task nodes given historical values and queue resource modeling.