IBM × Confluent: Is real-time streaming cool again?

Streaming is suddenly back in the spotlight—and in a big way—with IBM’s acquisition of Confluent, the company behind Apache Kafka, the open-source streaming platform that quietly powers much of the modern data ecosystem.

At first glance, this might look like a traditional enterprise company picking up a well-established open-source success story. IBM is, after all, more than a century old and no stranger to acquiring foundational open-source businesses like Red Hat and HashiCorp.

But this acquisition is different. Not because of who bought whom, but why.

Buried in the press release is a sentence that explains everything: “Building the real-time data foundation required to scale AI across every organization.”

That one line reframes the deal. Kafka isn’t suddenly hot again because of streaming for streaming’s sake.

It’s because AI has changed the stakes, and real-time data is now the critical missing layer for AI agents to actually work in production.

Let’s unpack that.

AI agents are here. But they’re flying blind

Scott Brinker (ChiefMartec) and Frans Riemersma (MartechTribe) recently published one of the clearest reports on the state of AI agents in real-world marketing environments. Their findings highlight a surprising reality:

AI agents are being deployed …

. . . but they’re not being fed the data they need to be intelligent.

A few numbers from the report jumped out:

Only 20.4% of marketers integrate transactions and browsing history into their AI agents.

Imagine a customer interacting with your AI shopping assistant, but the assistant has no idea they purchased something five minutes ago or just browsed three products this morning. The result isn’t "personalized"; it’s awkward.

And it gets worse:

Only 22.3% have integrated customer support tickets into their AI agents.

It’s hard to imagine anything more essential than knowing whether a customer recently filed a complaint, requested a refund, or had a frustrating interaction. If your agent doesn’t know that, it can’t possibly be empathetic or context-aware, two things customers now expect from AI-powered experiences.

The conclusion is obvious:

Companies are launching AI agents without giving them access to real-time customer context.

And that’s not a failure of AI models. It’s a failure of data plumbing.

Batch Is fine for analytics, but useless for agents

For the last decade, most customer data platforms, warehouses, and analytics systems were built around batch pipelines.

Once every 24 hours, data would sync, refresh dashboards, and populate customer 360 profiles.

For analytics, this was good enough. For AI agents, it’s not even close.

AI agents need to know:

• What did the customer do just now?
• What product are they looking at this second?
• Did they just open a support ticket?
• Did they abandon a cart 30 seconds ago?
• Did they click a link in the last email?

A 24-hour delay might as well be a black hole.

Real-time is no longer a “nice-to-have.”

It is the fuel that lets AI agents become contextually intelligent, trustworthy, and useful.

This is why IBM bought Confluent

Kafka is the backbone of real-time data at scale. Banks, retailers, logistics platforms, gaming companies–they all rely on Kafka to capture and propagate event streams instantly.

By acquiring Confluent, IBM isn’t buying “streaming technology.” It’s buying the distribution layer for AI.

AI without real-time context is static. AI with real-time streaming is adaptive.

IBM sees what many enterprises are now waking up to:

AI agents cannot operate effectively without real-time customer context, and Kafka is the foundation for that context.

This is the same pattern we saw when cloud took off: Companies that owned the underlying infrastructure became indispensable. Now, AI is creating its own infrastructure layer, and real-time data is at the center of it.

Real-time CDPs will become the AI nervous system

This is also why real-time CDPs like RudderStack are gaining traction.

It's not about collecting data anymore. It's about delivering the right customer state to the right AI agent at the right moment.

A real-time CDP:

• Ingests high-velocity events
• Creates up-to-date customer 360s
• Makes that state queryable via API
• Streams it into AI agents, LLMs, and workflows
• Ensures every interaction is context-aware and personalized

In other words: Real-time CDPs close the gap between what the customer just did and what the AI agent should do next.

And that gap, just milliseconds wide, is where all competitive advantage will live.

The streaming era was a preview. The AI era makes it essential.

Kafka spent a decade powering event-driven architectures, fraud detection, and log pipelines. Important, yes. But invisible.

AI changes that.

AI agents don’t just benefit from real-time data. They depend on it.

IBM didn’t just buy Confluent. They bought a critical piece of the AI operating system.

And as AI agents become the primary interface between companies and customers, real-time data stops being an engineering concern and becomes a CEO-level priority.

Put differently, what IBM is really betting on is customer context. Agent Data Platforms (ADPs) are emerging as the name for this stack: a real-time memory layer that keeps customer context up-to-date, paired with a decisioning layer that turns the context into action.

In a separate post, I go deeper on how ADPs relate to CDPs and why this memory layer matters for AI agents.

The message from the IBM Confluent acquisition is loud and clear:

Real-time data isn’t infrastructure anymore. It’s the intelligence layer of AI.

FAQs

What does IBM’s acquisition of Confluent mean for the future of real-time data?

It signals that real-time streaming is now core enterprise infrastructure. IBM is betting that AI applications require instant customer context, and Kafka is the backbone for delivering that context at scale.

Why do AI agents need real-time customer context?

AI agents must understand what a customer is doing right now to respond intelligently. Delayed or batch-refreshed data leads to outdated decisions, poor personalization, and broken AI experiences.

How do real-time CDPs support AI agent performance?

Real-time CDPs like RudderStack ingest streaming events, maintain always-current customer profiles, and make that state queryable for AI models and agent workflows, enabling context-aware responses.

What is the connection between real-time streaming and Agent Data Platforms (ADPs)?

ADPs extend real-time CDPs by adding a decisioning layer on top of the customer memory layer. They use live customer context to drive the next-best action for AI agents.

Is batch data still useful in an AI-driven stack?

Batch is still valuable for analytics, modeling, and historical insights, but it is not sufficient for real-time AI agents that need fresh behavioral data to act correctly.