Pictured: Ms. Majors near Linden Lab's erstwhile Brighton UK office. Yes, her shirt says "Will work for L$"
Charity Majors has an excellent blog post on creating culture and rituals within a tech company's engineering team, and knows whereof she writes. She's CTO and co-founder of Honeycomb, a leading observability provider — I.E., The service your company uses if it has a highly complex computer network and you want to see in real time across the whole system what's working well and what's not. Charity cut her teeth at Linden Lab — a company with an insanely, wackily complex computer network — and shares some of the insider rituals that the Linden engineers practiced, to keep up morale and keep Second Life online and running as smoothly as (relatively) possible.
Which brings us to Shrek Ears:
SHREK EARS
We had a matted green felt headband with ogre ears on it, called the Shrek Ears. The first time an engineer broke production, they would put on the Ears for a day. This might sound unpleasant, like a dunce cap, but no — it was a rite of passage. It was a badge of honor! Everyone breaks production eventually, if they’re working on something meaningful.
If you were wearing the Shrek Ears, people would stop you throughout the day and excitedly ask what happened, and reminisce about the first time they broke production. It became a way for 1) new engineers to meet lots of their teammates, 2) to socialize lots of production wisdom and risk factors, and 3) to normalize the fact that yes, things break sometimes, and it’s okay — nobody is going to yell at you.
So that's an inside insight for Second Life users. Whenever the virtual world was offline or constantly crashing, chances are someone in San Francisco was wandering around the office with green felt ears on their head. Those ears were worn, Charity tells me, "[L]iterally every outage."
But it wasn't a mandatory hazing ritual, she quickly adds: "The spirit of it was always very light hearted and fun. It wasn't to shame people, it was to congratulate people. You took down production? Congratulations, you're finally one of us now. Our thesis was that if you never took down prod, you probably weren't moving fast enough or taking enough risks. (Second Life was a crazy complex distributed system, even by today's standards, at a time before devops or IaaC or any modern tooling. The world was much more fragile one.)"
I remember Shrek Ears well, because wearing them began as a ritual during my own time at Linden Lab, from 2003-2006. In fact, they matriculated from the engineering team so that random people across the company would wear them after they goofed up in their own department. (I even put them on for awhile after I messed up a New World Notes detail, but I don't think anybody noticed.)
Speaking of Second Life downtimes, Charity told me about some major incidents from her time on the engineering team, from 2004-2010. Strap in, we're getting into the technical weeds, sharing a Shrek Ears photo, and talking about "Transactpocalypse" too:
(more…)
