Write once, use anywhere – the future of the cloud data warehouse

Today, we released a new product—Personalization API—that flips the role of the data warehouse on its head. Data warehouses have traditionally been associated with "back office" functions like analytics and reporting. For the first time, Personalization API brings the warehouse closer to the "front office" by offering a fast, flexible API to tailor customer experiences across web, mobile, and email.

What if you could combine the analytical power of the data warehouse with the real-time performance of a low-latency, globally distributed API? We've done it, and now we're offering it as a fully managed service, so you can too.

With a single click (and a SQL query), you can deploy data APIs that empower product and marketing teams to enrich customer-facing apps, dynamic email templates, and more.

The Hightouch Personalization API

If someone had predicted a decade ago that we'd be powering real-time production APIs using the data warehouse, I would have called them crazy. But in 2023, it's the obvious next step. This raises the question… how has the data warehouse changed from a technology standpoint? More importantly, how has its role evolved in the modern enterprise?

To understand where we're headed next, we need to briefly look backward.

The Early Days of the Data Warehouse

In the late 1980s, IBM researchers proposed the idea of a "business data warehouse," a central place for consolidating data and building reports so executives could make better decisions. Managing a data warehouse was a challenging feat back then. Data needed to follow a very strict schema planned well in advance with performance considerations in mind. If you needed to store more data than anticipated, or if you wanted to serve query results faster, you would need to enlist the help of database administrators and other specialized professionals.

Because of these heavy technical barriers, data warehouses failed to truly emerge as the central nervous system of the business. Instead, they were seen as a final resting place for data: an analytics-focused store accessible to only technical teams with the budgets and resources required to support them.

Emergence of the Cloud

In the late 2000s, the software landscape changed. First, the cost of data storage rapidly declined. Next, the rise of the cloud with AWS made compute resources far more elastic.

Instead of collecting and storing only the most valuable data, you could collect anything and everything. Instead of planning your server needs and required capacity months or years in advance, you could pay-as-you-go with models as granular as per-second billing.

Companies like Snowflake, Databricks, and Google Cloud (BigQuery) quickly realized the implications of these advancements for the world of data and analytics and delivered the cloud data warehouse. Cloud data warehouses and the ecosystem of tools that have emerged around them eliminated many of the administrative concerns from the first generation of data warehouses. They empowered data teams to focus on the most critical part of the value chain—delivering unique insights to the business.

The hardest problem in data

Write Once…Use Anywhere

This brings us to the last couple of years and an exciting moment in the story—because it's here that innovative business and technology leaders both did something unexpected. It dawned on them that the data in their warehouse was helpful for far more than just dashboards and reporting.

If you're old enough, you may remember the old Sun Microsystems slogan for the Java programming language: "Write once, run anywhere." The slogan highlights how a Java program is designed so that once it is written, it can be expected to run on any device equipped with a Java Virtual Machine.

The savviest people have realized that the warehouse has the potential to fulfill that same role, but for an organization's most valuable data.

Let's take a propensity score model in the data warehouse as an example. Suppose you want to use that for analytics or BI. In that case, the data warehouse is an obvious choice, but let's say you want to use it to deliver personalized emails, serve custom ad campaigns, or tailored content recommendations—what do you do then?

A traditional organization would have the email, ad, and website teams model the data (using slightly different information) and power their experiences separately. It's incredibly expensive, slow, and actually less accurate because it only uses a small subset of available data.

But a modern organization—well, they'd compute the score once in the warehouse, and then they'd expose the relevant data models to every team that needs it in a controlled and governed way.

Suddenly every team gets the benefits of the best analytics the company can bring to bear on a problem… and somehow, it's faster, cheaper, and better than the alternative. That doesn't happen very often in life and is a telltale sign that you're onto something.

Leverage warehouse data anywhere

This vision underpins our entire product strategy here at Hightouch and is exactly why my co-founders and I founded the company. We believe that the data warehouse disrupts how every part of a business will get access to and use data. And we built Hightouch to be a part of it.

The Role of Hightouch

I talk to customers at companies ranging from cutting-edge startups to Fortune 100 enterprises every day, and I have absolutely zero doubt that the "write once, use anywhere" future is coming.

At Hightouch, we see our role as providing the tools that make "use anywhere" a reality for every business. We call it Data Activation.

First, we built a Reverse ETL platform that can connect any structured dataset to more than 125 destination technologies. We actively help data and analytics teams serve thousands of marketing, sales, service, and product teams with data wherever and whenever they need it. Last year alone, we synced 1,923,349,425,886 rows of data from customers' data warehouses to advertising networks, business systems, and databases.

Next, we heard from our customers that the data in the warehouse is so valuable that their less technical counterparts wanted access to it for themselves. They needed a simple way to empower a typical Marketer that would still be governed and scalable, so we built the tools and features needed to enable organizations with a warehouse-native "composable" customer data platform that allows anyone to activate customer data.

Hightouch's audience builder unlocks the Composable CDP

This brings us to today.

A New Paradigm

As we continue into a new frontier of use cases powered by the data warehouse, we see software engineering challenges as the next big problem to be solved.

Historically, teams of software engineers have been needed to create the personalized experiences consumers expect, and businesses want to deliver. With the deployment of our Personalization API, we aim to make these once unorthodox use cases dramatically more manageable—again, leveraging the data warehouse.

For example, previously, to enable any product recommendation experiences, software engineers were forced to spin up dedicated application databases to store customer data before then making it accessible with custom-built endpoints. Now, with the Hightouch Personalization API, any data model—customers, products, transactions, recommendations, etc.—can be served instantly to any internal app, web experience, or customer journey management platform. Define your dataset with SQL, and Hightouch will handle the rest.

In a world where milliseconds matter, Personalization API is faster than you think. Under the hood, it's backed by a distributed, multi-region data store that delivers best-in-class performance, with our latest benchmarks showing p99 response times under 30 milliseconds in the same region. Our hybrid architecture allows you to process complex analytical models in your data warehouse and then serve those datasets to your customer-facing services conveniently and instantly over HTTP.

Here's an example of how a music streaming service can use the API to fetch user attributes, recommendations, and audience memberships from the data warehouse. The response can then be used to inject personalized content into emails, websites, and mobile apps.

Personalization API response

What's Next?

We believe in a world where every team in an organization can effectively use their customer data to deliver next-generation customer experiences and, ultimately, next-level business outcomes. And we believe the only way to achieve that vision is to build our platform on the data warehouse, the center of gravity for modern organizations.

In the coming weeks, we'll take this vision further as we release additional features that make activating your data more manageable, more accessible, and more powerful than ever in ways that are fundamentally impossible without the power of the warehouse.

Onwards, Tejas