We invested in Astronomer for two simple reasons – what it is today, and what it has the potential to become in the not too distant future.
Today, Astronomer helps enterprises optimally use Apache Airflow, the leading open source workflow management platform for scheduling data engineering pipelines. Modern applications are collections of many loosely coupled services, and a workflow is the proper orderly execution of these services. To programmatically author, schedule and monitor your workflows, and to be able to do all that in a dynamic and scalable fashion, you want Airflow. You can easily visualize your data pipelines’ dependencies, monitor the progress of each task and troubleshoot issues. And to do that as a managed service that is easy to use, flexible, powerful enough to handle complex DAGs, cloud-agnostic, and integrates seamlessly with the rest of your data stack, you need Astronomer.
What could Astronomer become in the near future? Well, one of the benefits of the modern data stack is the ability to combine best of breed solutions into your optimal individualized data architecture, but the truth is interoperability is and will always be a difficult challenge to solve. There is a crying need for a solution to manage and power the flow of data within an organization. With the release of Airflow 2.0 with a much better UI and a much faster and reliable scheduler, the general availability of Astronomer’s Astro cloud service and the acquisition of Datakin, the leader in real time, operational data lineage solutions, Astronomer is extremely well positioned to become the essential data processing orchestration company, providing complete data awareness and control to any organization: Airflow as the data plane, Astro as the control plane.
And why are we convinced this future state is so near and will be so meaningful? Look no further than the emergence of data engineers, arguably the fastest growing job title in the tech world today. These brave souls have an unenviable task – they must satisfy their colleagues’ voracious desire for data, by supplying all parts of the business with the exact right data in the right format at the right time, and do all this against a never ending rising tide in data volumes. What was once possible to accomplish using ad hoc custom scripts now requires declarative data pipelines and workflows. Put simply, the ability of a cloud scale company to generate insights from their data – to truly become “data driven” – is a direct reflection of their maturity in operationalizing the flow of their data assets. No longer does ETL stand for extract, transform and load – it stands for exist, thrive, and last. Yep, this is existential for your business. And your data engineer is the one keeping track of the nuclear launch codes. I think you might want to spend some money on Astronomer helping them...
So, ok, the answer to the “what could go right” question for Astronomer is intoxicating indeed. But it does beg an equally important follow-up question – does Astronomer have the right people in engineering and mission control to turn high potential into reality? That’s where Scott Yara and Joe Otto come in. We had the pleasure of backing Scott and Joe years ago at Greenplum, and heard their long term vision for turning an MPP analytic database into an open source, multi-cloud platform-as-a-service that would enable digital transformations at the world’s largest companies, and then, post an acquisition by EMC, watched them do exactly that. They don’t come any better than Scott and Joe at visualizing where the analytics world is headed, articulating that vision to the market, and then building the right product stack and the right GTM engine to capitalize on that future opportunity, and they aren’t alone. Leonard Walstad, Andrew Ettinger, Steven Hillion, Goutam Tadi, Mike LeBlanc, Nate Kapinos, Andrew Forbes, Carlos Castro, these are just a few of the experienced Greenplum/Pivotal operators that have already rejoined the team because working with Scott and Joe is just that compelling.
Big bold bets are, by their very definition, a challenge. Decades ago [shamelessly paraphrasing the words of President John F. Kennedy] we chose to go to the Moon not because it was easy, but because it was hard – and because “that goal will serve to organize and measure the best of our energies and skills.” It was a challenge we were willing to accept. Yes, owning the orchestration layer for the modern data stack is not even on the same difficulty scale as going to the Moon (don’t forget JFK spoke those words in 1962, when computers still used punch cards and Charles Bachman of GE had just developed the very first database management system), but we at Meritech believe it can be equally inspirational. Those of you who want to join a mission to measure the best of what you have, we have only one ask - spend an hour with Scott, Joe and the Astronomer team hearing their vision first hand. Just be prepared, odds are this will be an opportunity that you’ll want to take your own giant leap to be a part of.