I think Databricks understands that companies will always need different data tools in their stack for various reasons. Similarly, most companies are multi-cloud, even when they try so hard not to be. Maybe you have a long-running SAP contract, your risk team is using Palantir ontologies to prevent fraud, or your research lab is developing a Knowledge Graph on top of Altair’s technology. If Databricks Data Intelligence Platform can’t replace every other tool in the stack, how can it be important for all of them? That’s where the Data Lakehouse, Unity Catalog, and Open Data formats come into play. 

While companies are okay with using multiple Data & AI specialized tools, there is a requirement that these tools do not become data silos. So there is a push to “shift left” most of the data processing logic to a centralized data platform that can communicate seamlessly with all the different tools while keeping a federated data governance and security process.  The platform that can win this race has a lot to gain, even if a lot of the data storage and computing will remain distributed across multiple tools. 

While there is no hard lock-in that was common in the 1990s and 2000s monoliths, it’s evident that most organizations will want to keep their catalog centralized and be able to query and process all their data from different sources as easily as they can in a single platform. And that’s even a stronger need with all the AI advanced use cases that need high-quality proprietary data to make a difference in the marketplace.

Databricks is moving fast in this space to make its Unity Catalog the leading Data Catalog by releasing it open-source and building integrations with most enterprise vendors. There is a similar move here: integrating the two leading open data formats (Delta and Iceberg) into one format that Databricks develops. By enabling companies and vendors to use Unity and Delta/Icebergs as industry standards, Databricks is uniquely positioned to become the foundational layer for the Data & AI stack, the same way  NVIDIA’s CUDA became the universal format for AI development in the GPU. From this angle, one can understand why Databricks is fine with clients using Altair or Palantir for specific use cases that they may have been able to develop inside Databricks’ tooling. As long as the heavy lifting is in Databricks, they are winning the race.

There are other areas in the data stack that Databricks seems to be fine with customers using partners that excel in their niches, even though there is a Databricks-native alternative for it. dbt is a good example, helping Databricks clients develop their data Lakehouse in a standardized, platform-agnostic way. By connecting dbt Cloud to Databricks, customers can deploy their models faster, increasing Databricks consumption while keeping the data and metadata inside Databricks.

We at Indicium have always looked for partners who deliver the most value for our clients while maintaining the flexibility needed to build future-proof, world-class data platforms. Databricks‘ recent moves have been strategic and allow us to deliver real value with other top partners, such as dbt labs. 

If you want to understand how to build or improve your data platform with some of these vendors, contact us.