Jay Kreps

Jay Kreps is the software engineer and entrepreneur who, more than almost anyone else, turned "real-time data streaming" from an academic curiosity into standard enterprise infrastructure. As a principal engineer at LinkedIn in the early 2010s, Kreps led the team that built Apache Kafka, a distributed log-based messaging system designed to move enormous volumes of event data reliably between systems. Kafka became one of the most widely adopted pieces of open-source infrastructure of the decade, now running inside the majority of Fortune 500 companies.

In 2014 Kreps co-founded Confluent alongside two other Kafka creators, Neha Narkhede and Jun Rao, to commercialise the technology and build a managed streaming platform around it. Under his leadership as chief executive, Confluent grew Kafka from an open-source project into a public company, listing on the Nasdaq in 2021 and building a business spanning cloud streaming, stream processing (ksqlDB, Flink) and data governance.

Kreps is also a prolific systems thinker, having authored Voldemort (a distributed key-value store) and Samza (a stream-processing framework) during his LinkedIn tenure, and writing extensively on the "log as a first-class citizen" philosophy that underpins modern event-driven architecture. His essay "The Log: What every software engineer should know about real-time data's unifying abstraction" is considered foundational reading in distributed systems circles.

In 2026, IBM announced its acquisition of Confluent, folding the streaming platform into IBM's hybrid cloud and data portfolio; Kreps subsequently stepped back from the chief executive role after nearly twelve years running the company, handing operational leadership to long-time chief product officer Shaun Clowes. He has taken on a board seat at Anthropic, extending his influence from data infrastructure into the governance of frontier artificial intelligence development.

Kreps's legacy is structural: Kafka's publish-subscribe, log-based model reshaped how companies think about data movement, feeding everything from fraud detection to recommendation engines to the pipelines that increasingly train and serve AI models. Few engineers have had as direct an impact on the plumbing beneath modern data-driven business.