9/25/17
By Peter High, published on Forbes
What are microservices? In the first part in this series, I interviewed Matt Miller of Sequoia Capital. In this, the second part of the series, I interview Jay Kreps, the founder and CEO of Confluent, an open-source streaming platform based on Apache Kafka. Kreps developed Kafka while he was the Principal Staff Engineer at LinkedIn. Three years ago, he left that social media giant to develop his own company to develop real-time data streams leveraging microservices.
In this interview, he describes his entrepreneurial journey, highlights the opportunity that microservices offer to large and small companies alike, and offers advice on how best to harness the power of this trend.
Peter High: Jay, you were at LinkedIn for seven years; the last three as the principal staff engineer. You came to microservices through that experience, and then developed some key technology in the evolution of the topic. Can you talk about the opportunity as you saw it from your position at a fast-growing dynamic technology company?
Jay Kreps: When I joined LinkedIn in 2007, they were starting to figure out how to effectively break up a monolithic application that everybody put their code into. There had been an earlier wave of technology called service-oriented architecture, which is pretty much the same thing as microservices, that had died off. LinkedIn was born in the space between service-oriented architecture and the advent of microservices. Generally, to scale a software engineering effort, you add software engineers. However, when you add engineers you do get more done, but each individual engineer adds less capacity than the one before. This is a fundamental problem of big projects with lots of people and big applications; as you add people, you get slower. When people talk about microservices, they discuss scaling for more web traffic, reliability, and all kinds of other things. Microservices do not help those things.
Microservices only help one thing — scaling software engineering efforts. It lets you add more money and turn it into more software at a more constant rate.
LinkedIn made a lot of mistakes when they were trying to scale because not a lot was known about best practices. As a fast-growth company, one of the most important things was creating a product that could evolve quickly. Having agility and speed was critical because the social networking space was competitive. It ended up working out in the end because everyone could deploy their part of the application and move independently. However, there were a bunch of hills and valleys in between where we made changes that were supposed to make us more effective, but had the opposite effect.
A significant evolution in technology was needed to get the promised outcome. This is where Apache Kafka came in. People starting out today benefit from the knowledge of what works and what does not, and many of the tools around deployment and the communication between services are built up. There is now a whole family of technologies that solve problems around deployment, how you monitor your applications to make sure they are all running well, how they communicate with each other, and how you can ensure reliability and security globally when you have many moving pieces.
High: Apache Kafka is the open source data-streaming platform you co-developed during your time at LinkedIn. It is also part of the backbone of Confluent. What was the process of codifying that into open source technology?
To read the full article, please visit Forbes
