Zynga open sources homegrown server diagnostics tool

zPerfmon takes pulse of production systems to identify problems

12 August 2013 by Yevgeniy Sverdlik - DatacenterDynamics

Zynga open sources homegrown server diagnostics tool
Zynga's new CEO Don Mattrick (right) and Mark Pincus, the company's co-founder and former CEO

Engineering team of the online gaming firm Zynga has open sourced zPerfmon, a tool created in house and used to identify what causes problems in the infrastructure that serves the troubled company's shrinking yet still enormous (about 39m) user base.

 

As Binu Philip, a Zynga architect, described it in a blog post, “zPerfmon was conceived and built to answer one question: what went wrong in production?”

 

Zynga's production environment runs dozens of games and services. “While we can unit test and stress test and feature test, it can never cover all that happens in production,” the architect explained. “If something unexpected happens in production, which is all the time, and there are no footprints in the sand, investigation is arduous and frustrating.”

 

This is why the team built zPerfmon.

 

The zPerfmon server is a “processing engine with a heartbeat of 30 minutes,” Philp wrote. This means the server groups, slices and dices all data available to it every 30 minutes.

 

The data includes execution profiles from certain production servers – each with a schema to identify page, time, machine configuration and other details – the number of users and instances and system metrics.

 

Using timestamps, a user can drill down “from increase in instance count to a spike in CPU to a page which had a missing break in a foreach() loop,” Philip explains. The user interacts with this data through a web UI (user interface).

 

The solution also does things like generating scheduled reports, running scripts to compute headroom, cleanup and other maintenance tasks.

 

zPerfmon was built to run on one server, be very resilient and to not cause performance issues for the infrastructure it analyzes.

 

The code is now available on Github: https://github.com/zynga/zperfmon

 

Zynga is going through a rough period. Its revenue for the second quarter of 2013 was US$231m – down from $332m reported for the same period last year.

 

The number of daily active users of its games went from 72m in the second quarter of 2012 down to 39m in in the second quarter of this year.

 

The company has been regularly laying off employees and brought in a new CEO, Don Mattrick, in July.

CONNECT WITH US

Sign in


Forgotten Password?

Create MyDCD account

Regions

region LATAM y España North America Europe Em Português Middle East Africa Asia Pacific

Whitepapers View All