Send in the Clouds: Martech Moves to Cloud Platforms

Last weekend brought the intriguing rumor that Salesforce is in late stage talks to acquire data integration vendor Informatica.   This followed the previous week’s rumor that Google-parent Alphabet is consideringan offer for Salesforce-competitor HubSpot, and came the same week as a slew of partnership announcements tied to Snowflake’s Marketing Data Cloud Forum.

You don’t need a crazy wall to see how these events are connected.  Cloud database vendors including Google and Snowflake are expanding into marketing applications.  This poses an obvious threat to customer cloud vendors like Salesforce, in good part because it expands the ability of IT and data teams to build their own solutions rather than buying them.  Viewed in this light, Informatica helps Salesforce to compete by strengthening its ties with IT and data teams.  More strategically, an Informatica that is properly integrated with Salesforce Data Cloud would help those teams build more powerful databases within Salesforce.  This would make Data Cloud a more viable alternative to Google Cloud or Snowflake as the company’s primary customer data store.  In Salesforce’s wildest dreams, Data Cloud might even replace data warehouses.

Let’s put aside the question of whether Salesforce could or should become an enterprise data warehouse supplier.  The immediate question is whether it can retain its position as a primary platform for customer data and customer-facing applications, or that role will be taken over by cloud platform vendors like Google Cloud, Amazon Web Services, and Microsoft Azure and cloud database specialists like Snowflake and Databricks. 

I don’t know the answer.  But what does seem clear is that customer systems will run on platforms, whether from Google, Salesforce, Snowflake, or someone else.   

The platform approach isn’t particularly new but it’s not the way things have always been.  Remember that most customer systems – even today – are stand-alone, self-contained products that maintain their own databases.  These are the dreaded silos we keep hearing about.  The trick is to connect these silos, initially at the data level by copying their data into a central warehouse or Customer Data Platform, and ultimately at the decision and delivery levels through shared journey orchestration and messaging. 

Platforms solve the data problem, since all systems run on a common data store.  They don’t necessarily unify the decision and delivery layers, since those functions are handled by applications built on the platform.  Those boundaries are not clear: platform vendors sometimes add decision and delivery functions.  Indeed, the risk that the platform vendor will incorporate your application is one that application developers must accept as the price for accessing the platform vendor’s customers.  Data clean rooms are a good example: once independent applications, they are now offered as core platform services.  Machine learning and artificial intelligence are following the same path.

The transition from silos to platforms is far from complete, but the industry direction is clear.  The entry of cloud and cloud data platforms as alternatives to customer clouds is likely to accelerate the change.  As we’ve already seen, platforms are a mixed blessing for application developers: they gain access to a larger audience but risk of being undercut by a platform product extension.  (The good news is those extensions often involve acquiring a leading application.)  

Application vendors also benefit from the platform providing data management functions that the application vendor would otherwise have to build for themselves.  This simplifies application development, freeing resources to improve the primary application functions.  Unfortunately, it also makes it easier for other companies to build competing applications.  This lower barrier to entry (and to continued survival) is one reason the number of marketing applications has increased so sharply in the past decade.

And what does all this mean for marketers and other business users?  Since the platform model isn’t new, we can already draw on experience.  It’s a mixed bag: buyers have a greater choice of applications for any particular task, which is mostly good but does create a burden of having to choose.  Buyers are tightly bound to whichever platform they select, since switching platforms means switching the related applications as well.  This creates a quasi-monopoly situation, with the potential for higher prices and poorer service that comes with it.  Raise your hand if you’ve seen that already.  This will only become worse as platforms become more common, making it harder for independent application vendors to survive and, thus, harder for companies to assemble their own ‘best of breed’ architecture from separate point solutions.  In other words, you may miss those silos when they’re gone.

It doesn’t have to be this way.  The alternative to a platform architecture is a true composable architecture, where different applications are written to common standards that are not controlled by any particular platform.  The ecommerce world has managed this to some extent, with ‘headless’ solutions and standards enforced by the MACH alliance.  Standard data access languages like SQL are another example.  That the customer data world will come up with similar standards seems doubtful, although there’s always hope.   

It’s also possible that integration platforms will let different systems interoperate even without shared standards.  That’s certainly their goal but so far results are limited by the effort to accurately capture all the nuances of one system and present them to another without losing anything in translation. AI might well change things by making this translation easier.  But AI might change a lot of things, so let’s not count on any of them quite yet.

For the immediate future, then, we can expect a handful of platform vendors to control customer data at a growing number of companies.  Those companies will be captives of the platforms, but it will be a fairly enjoyable captivity, with many pleasing applications to choose from and few obstacles to working as they please.  Every so often they’ll pull back a curtain and see the bars on the windows. Some will be so unhappy that they'll escape.  But most will accept the limits of their chosen platform and get on with their work.

 

 

Composable vs Packaged CDP: How Can We Help?

“Composable CDP” has been consuming much attention at CDP Institute as we wrestle with how best to help buyers who are considering it as an option.  The Institute published a Composable CDP Self-Assessment tool a few weeks ago, which gives a checklist of the functions required to replicate the profile-building capabilities of a standard CDP.  This was intended as the centerpiece of our Composable CDP Knowledge Hub but has attracted fewer users than expected.   The explanation I find most plausible is that few people can answer the Self-Assessment’s detailed questions about existing systems and requirements. 

In itself, this isn’t terrible: much of the value in the Self-Assessment comes from giving users a comprehensive checklist of items to consider, thereby highlighting the true scope of a Composable project.  Still, it’s clear the Self-Assessment isn’t helping buyers as much as we had expected, so we’re considering what other tools might be more useful.

The discussions surrounding this have helped to clarify my own understanding of where Composable CDP fits into the greater scheme of things.  The key insight is that the choice to use a Composable or packaged CDP is ultimately a tactical decision that addresses just one stage of the CDP project.  It doesn’t affect the previous stage, which is requirements definition, or the following stages, which are deployment and activation.  To overstate the case just a bit, marketers and other CDP users don’t care how their CDP gets built; they just want to use the resulting profiles to do their jobs.  To the extent that CDP users do care about the Composable vs packaged decision, it’s because that choice impacts the cost, timing, and quality of their system.

This insight in turn clarifies the distinction between knowing enough to make the Composable vs packaged decision, and actually deciding which is the best choice.  The knowledge needed to make the decision includes:

• Defining business requirements, which requires business users who understand their needs. 
• Understanding existing systems, so you can identify gaps that must be filled to meet business requirements
• Understanding the organization’s capabilities for filling the gaps with either a Composable or packaged solution.  These capabilities relate to technical resources including staff skills and budgets.  I believe that assembling a Composable solution requires more extensive technical resources than buying a packaged CDP, although some Composable advocates might disagree.
• Estimating the costs of delivering a Composable solution and a packaged solution so you can compare them.

The Self-Assessment tool addresses only the first two of those items: it asks users what they need and what their current systems can provide.  So, even if the users could answer all the questions, it wouldn’t provide all the information needed to make a sound decision.  Again, this isn’t exactly a flaw, since those are two important types of information.  But it is a limitation.

Only after all the information listed above has been gathered can the company address the second question of whether Composable or packaged is its best choice.  Answering this question depends on the combination of capabilities and costs.  That is, for Composable to be a viable option, the company needs to have adequate technical resources to assemble a Composable solution, and for Composable to be the best option, it has to offer the best combination of cost and value.  

(If we assume that either option delivers the same value, then the only difference is cost.  In theory, every system that meets same requirements will deliver the same value.  But in the real world, there will be differences in the quality of the resulting systems.  These will depend on the specific tools that are chosen, not simply on whether the solution is Composable or packaged.  This means that buyers must evaluate specific alternatives for either approach.)  

Circling back to the original question of how the Institute can help users who are considering a Composable CDP, it’s clear that few people will have all the information they need available when they start the process.   So the best we can do is to provide a framework that identifies the four kinds of information to collect and, perhaps, provides a place to store it over time.   That could be as simple as a spreadsheet, which isn’t very exciting from a technical standpoint.  But if it’s what our members need the most, then that’s what we’ll deliver.

Addendum:  Taking things a step further, here's what the tables described above could look like.  It would be interesting to collect separate answers from business users and IT/data teams, who very likely would disagree in many places.  I could easily convert these tables into an online format and have the system do some light analysis, including a comparison of answers from business users vs IT/data teams.  But we'd still run into the same problem, that it takes time assemble answers.  So this pushes me back to a spreadsheet that people can fill out at their leisure. You can download that here.

1 & 2: Requirements & Existing Systems
Data Sources	Not needed	Needed and Fully Available	Needed and Partly Available	Needed and Not Available	Don’t Know
- Website
- CRM
- Data warehouse/ - data lake
- Email
- Third party data
- eCommerce
- Web advertising
- Social media advertising
- Point of Sale
Profile Building Functions
- Capture data
- Ingest data
- Prepare data
- Store data
- Link data
- Build profiles
- Share profiles
- Integrate profiles
- Segment profiles
Activation Functions
- Audience selection
- Predictive analytics
- Campaign definition
- Real time interactions
- Cross-channel orchestration

 

3. Resources
	Rating (1=low, 5=high)
- Customer data infrastructure
- IT/Data team: customer data experience
- IT/Data team availability for this project
- Business users: customer data experience
- Business user availability for this project
- Training resources
- Measurement resources
- Budget for this project
- Management Support

 

4. Cost
Hard costs (enter dollar amounts):	Composable	Packaged
- Licenses/Vendor Fees
- Labor/Development & Integration Costs
- Operations (hardware, software, on-going maintenance & upgrades)
Other Considerations: (1=disadvantage, 2=same, 3=advantage)
- Time to Value
- Delivery Risk (time, cost overruns)
- Performance Risk (scalability, performance)
- Roadmap (control over features)
- Security/privacy (control over data)
- Quality (best components, competitive advantage, vendor, usability/consistent interface)
-

Does CDP Need a New Definition?

The earliest Customer Data Platform systems were introduced before 2010; the term CDP was coined to describe this emerging class in 2013. My definition had changed very little when we launched the CDP Institute in 2016, and has been the same ever since: “packaged software that builds a unified, persistent customer database accessible to other systems”. The Institute added the RealCDP checklist* in 2019 to attach more specifics to the definition in the hopes of helping buyers ensure a system that called itself a CDP could actually support the use cases they expected a CDP to support. By then, industry analysts were beginning to offer their own definitions which, while worded differently, were broadly consistent with the Institute definition. Even the major marketing suite vendors, who initially argued a separate (“persistent”) database wasn’t necessary, eventually discarded that position and introduced products that matched our criteria.

A successful concept like CDP quickly takes on a life of its own. It soon became apparent that many people were using CDP in a much looser sense to mean any system that built and shared customer profiles. This extended past packaged software to include custom-built systems and included systems whose scope was more limited than a true CDP. This expansion skewed some survey resuls but otherwise seemed relatively harmless; in any case, resistance seemed both pedantic and futile. What really mattered was these systems still gave CDP users access to the unified profiles.

Unfortunately, the evolution of the term didn’t stop there. As CDP became popular, many vendors adopted the label whether or not they actually met even the looser definition. At the same time, legitimate CDP vendors offered additional capabilities to analyze and deploy the data in the profiles. The resulting confusion ultimately led some vendors to avoid the CDP label entirely because it no longer provided a useful way to differentiate their products. Today, vendors seeking the latest label are more likely to call themselves digital experience managers than a CDP, even if their products meet the CDP requirements.

But the greatest challenge to the utility of the CDP label arose in the past few years when a number of vendors chose to claim that the core feature of the CDP – a dedicated database built by importing data from other systems, a.k.a. “persistence” – could be abandoned while still calling the result a CDP.  Their argument was the customer profiles could reside in a general purpose data warehouse, which most companies already had in place. 

The claim gained some plausibility from the fact that modern cloud data warehouse technologies, such as Snowflake, Google Big Query, and AWS Redshift, are in fact used by some conventional CDP vendors. The problem was they implicitly assumed that every company’s data warehouse had organized the data into useable customer profiles. In fact, very few data warehouses perform the specific tasks, most notably identity resolution, customer-level aggregation, and real-time response, needed to support CDP use cases. While it’s technically possible to add those features to an existing data warehouse, it’s usually a major project that often costs more, takes longer, and delivers less useful results than installing a separate, conventional CDP. (As always, the details depend on the situation.)

One positive result of the interest in warehouse-based profiles has been the decision of some CDP vendors to break their systems into modules that let users buy the data preparation functions separately from the rest of the CDP. This lets companies that want a warehouse-based system to still benefit from the mature capabilities those CDP vendors have developed over many years. These vendors have also often added the ability to combine data from a warehouse or other external system with data stored in a conventional, persistent CDP database, without actually loading that external data into the CDP. This gains some advantages of the warehouse-based approach, such as reduced data movement and storage costs, while retaining the benefits of the persistent CDP, such as greater control and flexibility.

You’ll note that all these changes affect the input side of the CDP data flow. It was once a very simple process: data from other systems was copied into the CDP, where it was formatted into profiles and shared with other systems. Now, the input process may be some mix of copying data into the CDP, reading fully-formed profiles from a warehouse, or combining internal and external data. By contrast, the delivery side of the process has remained the same: the CDP shares its profiles with other systems. 

In some cases, this has led to a subtle shift in perception of the CDP’s purpose: from a system that builds customer profiles, to a system that delivers those profiles to other systems. In this view, the core function of the CDP is to convert general purpose profiles into the specific formats needed by analytical, orchestration, and delivery systems (which we can call “activation systems” if you’ll trade some jargon for simplicity). This may still involve some data processing, such as advance calculation of model scores and aggregates to make them available in real time, and new data structures to hold the results of that processing. So there’s more happening here than a simple data transfer or API call.

Some people carry this shift even further and argue the CDP should really be defined as an activation system that reads profiles from somewhere else. Since three-quarters of the CDPs we track actually do have at least some activation capabilities, this isn’t quite as crazy as it may sound.

All that said, I’m not yet ready to redefine CDP. “Packaged software” and “persistent customer database” are meaningful terms that distinguish one configuration from other approaches (“custom software” and “external profiles”) that can also make profiles available to other systems. More important, the packaged/persistent configuration has significant cost and execution advantages over the custom/external alternative. So it’s important to avoid blurring the distinction between the two approaches. 

 At most, I’d offer retronyms that distinguish the “packaged CDP” (packaged/persistent) from the “warehouse CDP” (custom/external). By definition, the “warehouse CDP” doesn’t build its own profiles, so the term seems to ignore the fundamental function of the CDP.  You might call that oxymoronic, if not just plain moronic. But if we slightly redefine the core function of the CDP to be delivery of profiles rather than creating them, we can overcome that objection and, perhaps, give the market terms it intuitively understands.

You’ll also note the shift from profile creation to delivery puts the emphasis on the delivery side of the CDP flow, which we’ve noted has been the most stable and applies to both the packaged and warehouse approaches. This lets us join the trend to redefining CDP as a profile sharing system.

In short, the key distinction is whether the primary customer profiles are built and stored in the company data warehouse or in a separate CDP database. It’s worth maintaining that distinction because one approach relies on the company’s technical staff to assemble the functions needed to build and store the profiles, while the other relies on a packaged CDP to provide those functions. There are variations within each theme, including whether the warehouse uses modules from a CDP vendor to assemble its profiles or to help deliver them, whether real-time data is posted to the warehouse or held separately, and whether the CDP enriches its profiles with external data without importing that data. These are important from a practical standpoint, but do not affect the fundamental architecture of the system. Focusing first on where the profiles reside should help buyers understand the most important choice they have to make. This choice will in turn determine the other decisions they have to consider. 

 

_________________________________________________________

* ingest data from all sources, retain all details, keep the data as long as desired, build unified profiles, share the profiles with any other system, and enable real-time event-triggers and profile access.

Do Self-Service Systems Really Lead to Better Results? Our Member Survey Offers Surprising Answers to Industry Questions

The CDP Institute just published its annual Member Survey, which is always a treasure chest of interesting data. I’ve already published my primary analysis on the Institute site (you can download it here) but wanted to call out a number of findings that either contradict or confirm martech industry conventional wisdom. After all, nothing’s more fun than tweaking the nose of authority.

Data unification is growing: false. Most of us have a deep-rooted confidence in progress, at least when it comes to technology (human nature is another matter). The need for unified customer data has now been so widely understood for so long that we just naturally assume that more companies will have developed it. That has been true since the survey began in 2017 through last year’s survey: both CDP deployment and presence of a unified customer database have increased steadily. But both measures fell in the current survey. It’s hard to imagine companies have actually abandoned their unified databases, but even if growth has only slowed, that would be a big surprise.

Budget pressures are slowing industry growth: true. Some industry vendors report business is booming, but most will admit buyers are taking longer to make decisions. Sure enough, the fraction of vendors who reported growth in CDP investment is down from last year’s survey, both for the past year and the current year. 

Budgets may not be the only reason for slower growth, but the budget pressures rose more quickly than any other obstacle (from 20% to 29%). Cooperation, which can also be a symptom of budget pressures, grew the second-fastest (36% to 43%). 

 

Budget-pressed buyers are making smarter decisions: false. I can’t point to anyone who has made this exact claim, but think it’s implicit in reports that companies have more martech than they need and hope to simplify their stacks in the future. 

The survey does show that budget pressures are changing martech selection methods: more are selecting on cost (up from 43% to 54% for operating cost and 42% to 51% for initial cost), while selection on feature sophistication and breadth have fallen the most (26% to 15% and 41% to 24%).

Unfortunately, our surveys have consistently found that selection on cost correlates with low satisfaction with martech investments, while selection on features correlates with high satisfaction. So it seems that a short-term focus on cost is likely to cause long-term problems with martech results.

 

Martech departments are growing: true. The fraction of survey respondents who said they work in a martech department has more than doubled since the last survey, which was the first that listed martech as an option. It’s unlikely that the number of people working in martech has actually doubled in the past year, but it does seem reasonable to believe that some meaningful fraction of employees have been moved from marketing or IT into a dedicated martech department.

IT staff is playing a larger role in martech decisions: true. Despite the growth in respondents who work in martech departments, the current survey showed a small increase in the fraction of respondents who reported that that corporate IT manages their marketing technology (28% to 30%) and sharp declines in the fraction reporting that a martech team was in charge (43% to 32%) or each department runs its own martech (27% to 18%). This may be another cost-saving measure or it may reflect the growing importance attached to customer data (and martech in general) throughout the enterprise.

The bad news is that IT responsibility also correlates with lower martech satisfaction. Bear in mind that survey respondents are themselves mostly martech and marketing people, who are generally happiest when their own team is in charge. IT people probably give a different answer but are a small fraction of the survey respondents.

CDP projects are easy: false. Past surveys have found that about 60% of deployed CDPs are reported as delivering value, while the remaining 40% are struggling. I have always suspected most of the 40% are new projects that will deliver value eventually. The success rate is much higher in the current survey (80%), possibly because the slowdown in deployment has meant there are fewer new projects. But I'd want to see similar results in one or two other surveys before accepting that as a trend.

A separate question, asking vendors about success rates, finds the fraction reporting that say almost all or the majority of projects are successful has grown from 48% to 54%. While this might suggest some improvement in success rates, the more important message is that a bare majority of vendors say most CDP projects succeed.  Even when answers from CDP vendors are tabulated separately, just 68% say nearly all or a majority of projects are successful. Figures are lower for service providers (45%) and other respondents (15%). 

It's important to realize this is no worse than success rates for other large system deployments.  In fact, it's apparently better than average, since most studies put failure rates at 60% to 70%.  (See this page for a compendium.)  But these findings should dispel any notion that CDP deployments are easy. 

CDP projects fail because of technical complexity: false. It's also important to recognize that CDP failure rates are not due to any inherent problem with CDP technology.   As in previous surveys, by far the top reason for CDP project failure is organization. This has grown even more prominent in the current survey, while problems with poor requirements and CDP performance have fallen.

Comparing CDP status with satisfaction offers additional insight.  The satisfaction measure reflects success with martech in general, not with CDP in particular.  In other words, companies with a high score are "good at martech".  So, while it's not surprising that satisfaction rates are high among companies with a successfully deployed CDP and low among those who have not, this does support the position that CDP success is based more on the skills of the organization than CDP technology in particular. 

 

Privacy is growing more important: true. Last year’s survey showed a distressing decline in the priority given to data privacy regulations, with the share of firms making little effort to comply growing from 12% to 20%. That trend has now reversed, with the share of companies using privacy as a selling point increasing from 21% to 27%. 

Privacy was also listed in the CDP benefits question for the first time.  It was cited by 22% of respondents, ranking seventh of eleven items, and shows a below-average satisfaction score. This may indicate that most CDP users are looking elsewhere for their primary privacy management solution.

 

Self-service leads to success: false. The martech management section of this year’s survey added a new question about whether companies seek systems that empower business users to execute tasks without technical assistance. The concept of “no-code” is directly relevant here, although we didn't use the term.  This capability was by far the most common management technique, which wasn’t surprising: “empowering end-users” is a popular goal that saves money and makes users more effective.   But it also correlated with a low satisfaction score, which was surprising indeed. Is it possible that self-service doesn’t save money or make users more effective after all?

 

Answers to another survey question shed a bit more light. Among the options listed for CDP capabilities were self-service data extracts and self-service predictive models. Self service extracts were a common requirement (41%) correlated with a roughly average satisfaction rating, while self-service models were an uncommon requirement (8%) correlated with a very low satisfaction rating. My interpretation is that self-service extracts are a simple task that’s well understood by business users, so companies with well-run martech operations provide that capability.  By contrast, self-service predictive models are a complicated task that few users are equipped to handle on their own, so they are prioritized largely by companies with a poor understanding of what makes for martech success.  The larger message is that self-service should be deployed only when users are ready for it – and pushing beyond those limits can cause problems despite the apparent savings in cost and time.

 

Real-time processing is a high priority for most users: mixed. Real-time processing is commonly cited as an important CDP capability. In fact, the CDP Institute’s RealCDP requirements include real-time access to profiles and real-time event triggers. Again looking at the capabilities question, we see that real-time profiles are indeed a common requirement (42%), while real-time recommendations are much less common (15%) but correlate with very high satisfaction. The lesson here is that there are different types of real-time processing, and users consider some more important than others. Discussions about real-time should include similar nuance.

CDP must load data from all sources and retain full details: mixed. Both of these items are on the RealCDP requirements list. But loading data from all sources is the highest ranked capability (78%) and correlates with roughly average satisfaction, while loading full detail is cited by just 14% of respondents and correlates with much lower satisfaction. As with real-time and self-service, I take these answers to show that users have a mature understanding of what they do and don’t need from the CDP.  Loading all data sources is a core CDP promise and goes back to the problem CDP was originally designed to solve: getting access to all customer data. Storing full detail is not needed in most situations.  In practice, users choose which data elements are worth the cost. That said, I still believe that users want the option to store any particular details they need.

CDP users want to access their data warehouse directly: false. This year’s survey added a capabilities question about reading data from an external source without loading it into the CDP. This is a hot topic in the industry, both as a way of supplementing a traditional CDP (which maintains its own data store) and as a way of building a CDP-equivalent system that relies only on data in the enterprise data warehouse. Only a small fraction of respondents (14%) were interested in this and that group reported exceptionally low satisfaction levels. I interpret this to mean that, despite all the marketplace noise, there is actually very little interest among knowledgeable users in building a CDP that relies primarily on external data.

Summary

This report contains more bad news about industry growth and budget pressures than I like to see, but the tech sector’s troubles are well known and it’s not surprising that CDP projects should suffer with everyone else. I’m especially reluctant to highlight the data about success rates, because I know it will be taken out of context by vendors eager to promote alternatives to CDPs. I should stress again that the main obstacles to CDP success are organizational, not technical, and that success rates reported here are consistent with tech project success rates in general.  Bottom line: CDPs are major projects that don’t always go smoothly but are not more failure-prone than any other major IT effort.

What worries me more than bad news is the hints that companies are making poor decisions. Prioritizing cost over requirements will surely lead to more companies purchasing unsuitable products. Putting IT in charge of martech will almost surely lead to unhappy martech users. While budget issues are unavoidable, poor management decisions are unforced errors. Your company should work hard to avoid them.

All that said, the most interesting results are the ones that challenge conventional wisdom. Do self-service systems really lead to bad results? Is the importance of real-time overstated? Is there really so little interest in reading data directly from a warehouse? These are headline topics in martech today, and are often accepted as truth with little discussion. The questions are worth asking because the reality is almost always more complicated than the simple answers. When is self-service useful and when is it over-used? What kinds of real-time processes are really important? How is external data best integrated with a conventional CDP? Answering those questions will help companies make better decisions and ultimately lead to greater martech success.