7 years @ Mozilla

A little more than 7 years ago today, I was at a park in Aliso Viejo with my then 2-year old son when @dan_portillo called. Mozilla was looking for a Network Engineer.

A little more than 7 years ago today I was interviewing in Mountain View. I carried a cell phone that could only make phone calls.

7 years ago today I walked into Mozilla’s Landings office and configured my own Thinkpad laptop by myself (we didn’t have Desktop Support).  

Two weeks later Mozilla’s company-wide All Hands fit in the back of Tied House. Two months later I moved my wife and two babies from Orange County to Northern California.

Along the way I bought a house and called this home.

7 years.

7 Years of Change

I’ve gone through 3 CEOs & 3 bosses. My team’s grown from 3 to 80. I manage an IT Operations team. I’m learning how to lead, how to lean into my strengths.

I’ve learned to delegate, to trust, to let go. And I’ve been surprised and impressed at every step by the caliber of people who chose to surround me.

I now have co-workers friends whose first names I don’t know and who I’ve only met in person less than a handful of times.

Before Mozilla I had only traveled internationally once. For my honeymoon.

I’ve been to Argentina (pictures) & China twice. Stepped foot in Peru. Been to Whistler twice. I’ve seen a bear eat out of a trash can and stood on top of a mountain in July. While it snowed.

I’ve been to Berlin, Switzerland, Amsterdam, Vancouver. Lost count how many times I’ve been to Toronto, to London, to Paris

I traveled to Nice and stayed with a fellow Mozillian. Half my twitter stream is in Spanish. My arguably best friend lives 8500 miles away.

Along the way I lost my passport.

7 years ago my first smartphone was a Palm Treo running Windows. I now carry three phones & a tablet and saw the future I want last year as Mozilla shifted to Firefox OS. I have — and want — the web in my pocket.

7 Years of Personal Change

My 9 month old daughter is now 7 (or 13, it’s really hard to tell). My son knows how to find my name in about:credits.

I’m deeply focused on being an advocate for Mozilla Webmakers. I care about eliminating the opportunity gap through Year Up. I’ve learned that a mission with purpose means more than profit.

7 years ago I was a shy introvert. I still am but you probably wouldn’t know it.  7 years ago I was a Windows user.

I’ve become a San Francisco Giants fan & saw my team win the World Series. Twice.

In the past 7 years I’ve made friends

Along the way I learned that without music life would be a waste

Celebrating 7 Years

I don’t want any gold watch.

I want to do the same thing I’ve done every day for the past 7 years. I’m going to go to work.  

I’m going to go and work with people who make Mozilla not just a job, who have made this my home for 7 years, and who are as passionate about Mozilla & the Open Web as I am.

recovering from an email outage

If I could do this week over I would.  Too bad I can’t.

Email today is vital.  Not having it makes your heart palpitate. 

Monday morning, during a swap of a failed hard drive (something we’ve done countless times) the storage array we use for email went offline.  The whole thing.  And for various reasons, the last known good backup was from awhile ago. 

I painfully remember thinking “oh shit” when I realized what this meant.

[This isn’t a post about all the things I should have done to make sure I was never in this spot.  Everything’s obvious now.]

I learned a couple things this week:

  1. Hire the absolute best people (and geezus, hire people smarter than you!). You never know when you’ll need them.  You never know who will have the answer to the problem.  Hire people who care about each other.  You never know when you need them to look out for the one guy who, in 73 hours, forgot to sleep.  The same one guy who has to run point on The Next Big Step in 7 hours.
  2. Work somewhere where everyone realizes we’re all fighting the same fight. I’m surrounded by coders and when we needed coding, 1492 python coders lined up to help.  Not a single one of them reports to me.
  3. Get upset, yell, demand results.  But realize when it’s the right time to yell and when it’s not.  During a firefight, I need you to be on the best fucking game of your entire life.  It is not the time to be berating you.  It’s the time to treat you like a hero, a magician.  It’s when I do what you tell me to do for you.
  4. Communicate the heck out of everything.  Throughout this outage we found other tools to use to let users know what was going on and what to expect.  I’d post updates even when the information I had was incomplete.  I’d say so.  I hated having folks in the dark.  
  5. Expect criticism.  Some of it will be searing.
  6. Realize that the people working under me on this are collectively smarter than I am.  Offer help whenever but let them work.  Take point at handling communication.  Make sure #5 doesn’t get to them. Remind yourself of #3.

It took nearly two days to get things back to an okay state, a state where we had new emails.  Still recovering data from backups and reconstructing state from a now corrupt MySQL database.  

I’ll probably never be able to express my gratitude to the team I manage for their efforts this week.  Sucks we got here but without thinking, I’d go to battle with this team again.

We made mistakes that got us here but we can talk about that later and make sure it doesn’t happen again.  


(This post is pretty delayed.  Had hoped to post this shortly after I got back but life happened and here I am a month later.)

I just got back from a little more than a week in Paris. I went mostly for work but since I’ve never been to France and only to Europe three times before, I tried to mix in some personal time.

One of my former colleagues whom I admire often blogs to write down his own experiences. I think that’s a neat idea and I find myself taking random notes on my phone while traveling now.  This post is a result of those notes.

I’ve never been to New York City but I imagine it to be much like Paris. A city that seemingly doesn’t have a bed time. I never got a hang of the time shift from California and would often be awake at 2am. So was the street outside the office. And so were all the restaurants around the office. The energy at 2am was as electric as the energy at 4am.

Walking around the corner at 2am to grab a crepe did not seem out of the norm. Mostly, I got to see things I had only seen in Ratatouille.

Spent a lot of time in Mozilla’s Paris office upgrading some of the network equipment and trying to understand what it means to work remote from Mountain View.

The time shift off California was something else. I’m surprised anyone at Mozilla Paris (or Europe in general) is able to function with Mountain View. 9 hours made real time dialog really difficult and really emphasized the need to record various meetings for time-shifted viewing.

I took a train down to Nice and met Cedric, one of the localizers. Nice felt like Santa Barbara in a lot of ways but with warmer water.

And here’s where Buenos Aires left an indelible mark on me. After my trip to Buenos Aires, I picked up an interest in a number of groups, including Gotán Project. They were playing at Les Nuits du Sud in Vence, just outside of Nice.

This was one of my trip’s highlights, especially when they played Santa Maria. I emailed a friend right after the show and commented to her,

Gotán Project was better than I imagined live. Such a great show. I mean I’m in France standing in a town square listening to Spanish music sung by an amazing woman with half the crowd dancing Argentine Tango. Everything I like about classical Spanish guitar & electronic music.

The most interesting thing about this trip was the number of non-Paris based Mozilla folk that kept arriving in Paris.  The first week I was there David Ascher was in town.  The following week, the US-based Jetpack team was in town. 

Paris certainly ranks in the top handful of favorite cities but still has a bit of a way to go before it can oust San Francisco.