on leaving mozilla

Mozilla IT 2012

Mozilla IT 2012

Last Wednesday, July 31, I left Mozilla as a full time paid-staff.

Seven years at Mozilla has fundamentally changed how I operate. It’s redefined my values and helped define my personal mission & purpose.

The impact I had on Mozilla and on the Mozilla Project extended far outside Mozilla IT Operations and I hope to continue to be connected to the Mozilla Project (but more on that in another post).

For now I’ll simply share the note I sent to Everyone@Mozilla:

Adios, au revoir, sayonara

On March 15, 2006 I quietly joined Mozilla. Seven years and four months later, I quietly take my leave.  July 31 will be my last day here.

I had this moment of clarity awhile ago when I realized I have an awesome set of leaders under me and it was time, as Mitchell has said, to let them step into new roles and for me to step out.

To my Mozilla IT:

I have enjoyed being there for you.

I have enjoyed seeing you grow.

I have enjoyed seeing you handle some crazy, complicated challenges.

I will miss the camaraderie we developed and shared with each other.

I will miss the late nights working with you.

You always worked with me, never for me. And as much as I tried to help you, you helped me.

For this, and much much more, I thank you.

To my friends,

I made friends with people I worked with, people who are friends first and co-workers second.  I will miss seeing you every day. This is probably the hardest realization.

You have helped me grow and become who I am today.

To All@Mozilla,

In many ways, you, and this organization, have made an impact on me that cannot be undone.

My thoughts and actions have been shaped by what we have collectively been accomplishing and what we have set out to accomplish.

I could write much much more, of course so I’ll just leave with this:

“Parting is such sweet sorrow that I shall say goodnight till it be morrow.”

Zimbra & Mozilla email, 4 months later

Four months ago was a very tough time in Operations. We suffered a catastrophic disk array failure on Mozilla’s mail server (I blogged about it too). A series of mistakes kept email offline for two days. This was the worst I’ve ever felt, both professionally and personally.

Fast forward to today. So. Much. Better.

We learned. We researched. We re-organized ourselves. Much like The Six Million Dollar Man, we rebuilt it better.

justdave posted his account, “Re-imagining Zimbra email at Mozilla” but I wanted to add my own color.

Background

During my interview at Mozilla in 2006, I was asked a bunch of questions about Zimbra. First I had heard of it. By the time I started I had learned quite a bit about Zimbra. Back in 2006, mozilla.com email was hosted externally and we began the process of moving email back in house. The company hosting email couldn’t provide SSL and wasn’t doing all the groupware things we needed.

Post Zimbra-gate (December)

I was mentally done with email. I looked at simply outsourcing. I looked at hosted Zimbra, hosted Exchange, hosted whatever.

We Mozillians, we’re a unique group.

  • We want to use the IMAP client of our choice. Some of us just want to use the web interface. Others prefer Microsoft Outlook. Or Thunderbird, or Mail.app or Postbox or mutt or pine or Sparrow or …
  • Calendaring is just as complex.
  • We need to support a wide number of mobile devices – iOS, Android, Blackberry, devices that support Microsoft’s ActiveSync – with both email and calendaring.
  • Some use Zimbra’s document sharing/storage
  • We need something that supports IMAP, ActiveSync, CalDAV, CardDAV.

We looked at what others at our scale and beyond our scale use for email. Oracle uses Zimbra. Comcast uses Zimbra. At. Scale.

We talked to others hosting their corporate email with Google Apps (and their 15-person staff managing their Google Apps mail!). We learned that deploying Exchange requires a move from OpenLDAP to Active Directory and a particular skill set that we don’t have in house.

Moving Forward

This incident highlighted the need to have a team focused on infrastructure. Our primary focus (and priorities) always tend to lean towards various Mozilla web properties or developer services.

So we did two things –

  1. Broke up a fairly flat Operations group and created an Infrastructure Operations team (and a couple others) to focus on services like email & LDAP, to name a few.
  2. Built a new environment for services that, when break, cause work stoppage, cause a line to form behind my desk. This Hyper Critical Infrastructure, or HCI, is isolated from the rest of the production environment, has different change control processes and is meant to hit as many “9s” as we can hit. It’s a very different way of planning than we had done in the past.This technology stack uses more corporate/enterprise technology than we’re used to using at Mozilla.

HCI Today

HCI straddles two high density, (~15kW) racks. It’s only relation to the rest of Mozilla production network is two 10GbE fiber drops from the network core.

HCI has it’s own Juniper SRX 1440 firewalls. Its own Juniper EX4500 switching. Its own NetApp FAS3270. Its own 5 node VMware ESX cluster, each machine having 2x 6-core Xeons & 192GB RAM.

In a couple months, services here will be replicated to SCL3 using various NetApp & VMware technologies.

We had planned to have HCI in production by the end of February but no one wanted to rush this (plus someone decided to have a baby).

Instead we slipped that to the last week of March and I’m glad we did. We consulted with Zimbra and others. We sent Desktop & InfraOps to training. We tuned and fine tuned.

Zimbra Today

We have mailboxes spread across seven mailbox servers and understand the metrics we’ll use to determine when to add more mailbox servers.

We migrated 1002 mailboxes from San Jose to Phoenix without anyone noticing, without any user impact, in just a couple days. In fact, we didn’t mention it until we were done.

We have instrumentation and trending and alerting on everything we could think of.

What’s next?

All is for naught without learning. We learned a lot and we’ve changed how we operate as a team.

Once bitten, twice shy.

recovering from an email outage

If I could do this week over I would.  Too bad I can’t.

Email today is vital.  Not having it makes your heart palpitate. 

Monday morning, during a swap of a failed hard drive (something we’ve done countless times) the storage array we use for email went offline.  The whole thing.  And for various reasons, the last known good backup was from awhile ago. 

I painfully remember thinking “oh shit” when I realized what this meant.

[This isn’t a post about all the things I should have done to make sure I was never in this spot.  Everything’s obvious now.]

I learned a couple things this week:

  1. Hire the absolute best people (and geezus, hire people smarter than you!). You never know when you’ll need them.  You never know who will have the answer to the problem.  Hire people who care about each other.  You never know when you need them to look out for the one guy who, in 73 hours, forgot to sleep.  The same one guy who has to run point on The Next Big Step in 7 hours.
  2. Work somewhere where everyone realizes we’re all fighting the same fight. I’m surrounded by coders and when we needed coding, 1492 python coders lined up to help.  Not a single one of them reports to me.
  3. Get upset, yell, demand results.  But realize when it’s the right time to yell and when it’s not.  During a firefight, I need you to be on the best fucking game of your entire life.  It is not the time to be berating you.  It’s the time to treat you like a hero, a magician.  It’s when I do what you tell me to do for you.
  4. Communicate the heck out of everything.  Throughout this outage we found other tools to use to let users know what was going on and what to expect.  I’d post updates even when the information I had was incomplete.  I’d say so.  I hated having folks in the dark.  
  5. Expect criticism.  Some of it will be searing.
  6. Realize that the people working under me on this are collectively smarter than I am.  Offer help whenever but let them work.  Take point at handling communication.  Make sure #5 doesn’t get to them. Remind yourself of #3.

It took nearly two days to get things back to an okay state, a state where we had new emails.  Still recovering data from backups and reconstructing state from a now corrupt MySQL database.  

I’ll probably never be able to express my gratitude to the team I manage for their efforts this week.  Sucks we got here but without thinking, I’d go to battle with this team again.

We made mistakes that got us here but we can talk about that later and make sure it doesn’t happen again.  

buenos aires

Last week, for eight days, I was able to step outside my normal role managing Operations and wear an entirely different hat.

I had an amazing opportunity to interact with the vibrant Mozilla and Open Source community; I got to interact with those who I help from the shadows every day.

I’ve had a couple days to let my thoughts soak in.

Something about Argentina and Buenos Aires resonated with me in a way that’s hard to describe. It is, perhaps, the first time in my life I’ve had a sense of reverse home sickness. From San Telmo to La Boca Caminito to Palermo to Recoleta Cemetary and Calle Florida, Buenos Aires oozed of culture. From pizza to empanadas to more gelato than I can remember, it’s unlike anywhere else I’ve ever traveled.

Persicco, best gelato around.

Persicco, best gelato around.

The bustling energy of San Telmo’s Feria de Antigüedades was matched only by the energy of the open source community I met.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

On Wednesday night I had the pleasure of attending the Firefox 4 Party. This was amazing. The night before I met with several of the event organizers, including Guillermo Movia. They were expecting 50 or so and instead had 150 at the party. I don’t know how to really describe what it was like, walking around and mingling with everyone (in my broken Spanish no less), hearing everyone talk about Firefox and Mozilla.

Thursday night we attended the first Hacks/Hackers MeetUp in Buenos Aires at AreaTres. The discussion was all in Spanish but I mostly kept up. I was amazed at the turn out. Was a far larger group than I would have imagined and made me realize how large the open source community in Buenos Aires is.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

Friday wrapped up with a Design MeetUp at Urban Station that Tara led. This turned out to be one of the surprise highlights mostly because of the discussion afterwards. I don’t often get to interact with the community in such an intimate venue and speak Mozilla.

You can take me out of networking but you can’t take networking out of me. At each place we went to I’d always check to see who I had upstream connectivity from and what my path to Phoenix or San Jose looked like. Urban Station had the quickest Internet I had experienced while in Buenos Aires.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades


On a personal side, since all of these events were after 6pm local time and I was shifted 4 hours off California, I found a lot of time to explore and soak in Buenos Aires. I walked more than I can remember, slowed down more than usual to look and listen. Ate. Indulged. Walked. Explored. Saw a ballet show at Teatro Colón. Went to a Tango show. Walked to Carlos Gardel’s house (Casa Museo Carlos Gardel) in Abasto. Inadvertently walked to Palermo and had mint iced tea. Bought a crappy umbrella and walked in the rain.

No doubt I was lucky to have a fantastic travel companion (I’ve thanked you, haven’t I Tara?).

Lastly, I want to share this:

I went with very little expectations and maybe a little nervous anticipation. I came back with a profound sense of Mozilla, of the community that supports Mozilla and a feeling of renewed purpose for why I work at Mozilla. I came back with more friends than I left with, with a twitter feed half in Spanish.

John Lilly used to talk about about great companies vs good companies. How great companies last; they may change but their mission remains. Mozilla, he argued, was on a path to be a great company. Today, the vehicle for Mozilla’s mission is Firefox. Tomorrow it could be something else. But the mission will remain.

This is the sense of Mozilla I was left with when I landed in San Francisco. The emotional connection people make with Mozilla, and more precisely, its Mission, is what will make Mozilla one of the great companies.

I’ll leave you with a couple pictures I took. Tara did a better job taking photos than I did – you should check out her Flickr gallery.

Community Events

Mozilla, Firefox, “fixing computers” & a room full of kindergartners

My son has a vague concept of what I do at work. He knows I work at “Firefox”, knows the dino and knows I “fix computers”.

He knows that if he wants to get online he has to double-click on the Firefox icon and then on the Kidzui icon:

kidzui-toolbarHe asked me if I could come to his school and show people how I “fix computers”.

I started off by asking if anyone recognized the logo on the back of my shirt.  Even to a room full of 5 year olds, the Firefox logo was instantly recognizable – there wasn’t anyone who didn’t know what it was.  I talked briefly about what Mozilla did (“we make a web browser”) and that I help fix computers when they break.

I thought about showing them Mitchell’s Mozilla Tree but probably couldn’t have done as well as Mitchell could have!

Then I pulled out my laptop and showed them Firefox (Minefield really).  I was in the middle of showing them Firefox and how what my son does when he wants to get on the Internet… and Minefield crashed.  Which was a great segue into “lets go look at these computers I brought and how I fix them!”

I brought three old Celeron “servers” (you can hardly call a Celeron a server) with lids off and we spent the next 20 minutes taking apart the machines.  We took out the computer’s brain and the fan to keep it cool (they didn’t believe me that you could cook food on the CPU when the computer was “thinking hard”).  We took out the two memory sticks and the hard drive and the IDE cable.

These are things most parents in their right mind wouldn’t do with their home computer and these kids really enjoyed physically touching these parts and asking questions.

Mary Colvig helped me gather up a bunch of Firefox bags and my two kids and I had stuffed stickers into each one. Kids went crazy over the Foxkeh stickers and there was a collective “awwww!” when I showed them the “don’t hurt the web” stickers!

Anyways, good times. I enjoyed talking about Mozilla in a very different setting than I’m used to.

I’m going to go work on my new role in community outreach now…

my face is cold

NSID 2008 had to come to an early close for me today. I leave Monday for my sister’s wedding in Chicagoland and while I would have loved to hold out till the end of the month or even a few more days, that just wasn’t realistic (something about looking presentable).

You could argue I could have waited a couple more days but shaving at this scale is best done at home and not in a hotel!  Too bad too because it was just starting to get comfortable.

Here’s to next year!