Scaling lessons from Googles CIO

Im here to share a little disaster porn with you.

And with that comment, Google CIO Ben Fried kicked off his keynote at the Surge conference and shared two essential lessons for folks trying to build systems at scale. Frieds keynote went beyond sharing his failures, though, all in an effort to help developers realize that the entire culture of IT must change.

The Industrial Age, when specialization ruled, is over, he said, and the focus should now be on having an end-to-end understanding of systems. Generalists, not specialists, are what matter when folks are trying to push the boundaries of what is possible. We scaled up to deal with our own success and the way we scaled up was through specialization but understanding how everything works is really important, Fried said.

Fried described a time before his role at Google, when he led the development of a trading platform for Morgan Stanleys high-value clients, and the series of mistakes the team made. But as he deconstructed what went wrong, he realized that one of the biggest issues was that until the product fell apart, the many, many people who helped build it had never been all in the same room before and had no real idea how the system was supposed to work. That was lesson one.

The second lesson was his realization that he and his team thought they could use smart software and some impressive infrastructure to deliver an application that would seem like a desktop app, over the less reliable and untrusted network that is the public Internet. We made fundamentally unsolvable problems seem like solved problems, he said. And his team believed it, until it all started unraveling.

Taken together, Fried explained that developers need! to crea te a new culture built on general knowledge and realization that when you are building at scale, you are at the edge of the known world. Much like you would pack differently for a cruise than for an arctic exploration, those building at scale need to realize that they arent on the Loveboat and are instead like Admiral Byrd aiming for the North Pole. Explorers need to understand something about a lot of things so they have a sense of who to call and what to do when problems arise.

In the question and answer period that followed, Fried elaborated on these concepts, telling someone that IT generalists are probably born, not made. He said at Google, the company looks for folks that want to keep improving their skills, and even has a program to help give those people the tools to be better engineers when they find those traits in employees. He said the Google culture is one where the general engineers who understand the system have a lot of input and power, which is a cultural shift that organizations that want to build at scale should try to implement.

Finally, he emphasized that while change may be the root of all evil for a developer trying to build out complex web systems, its also inevitable and the reason they all have a job. Developers must expect change and adapt to it rather than assume thats a solvable problem. The session went from disaster porn to a defining philosophy of the new developer culture, one that turns its back on the Industrial Age to embrace the webscale age.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.