recovering from an email outage

If I could do this week over I would.  Too bad I can’t.

Email today is vital.  Not having it makes your heart palpitate. 

Monday morning, during a swap of a failed hard drive (something we’ve done countless times) the storage array we use for email went offline.  The whole thing.  And for various reasons, the last known good backup was from awhile ago. 

I painfully remember thinking “oh shit” when I realized what this meant.

[This isn’t a post about all the things I should have done to make sure I was never in this spot.  Everything’s obvious now.]

I learned a couple things this week:

  1. Hire the absolute best people (and geezus, hire people smarter than you!). You never know when you’ll need them.  You never know who will have the answer to the problem.  Hire people who care about each other.  You never know when you need them to look out for the one guy who, in 73 hours, forgot to sleep.  The same one guy who has to run point on The Next Big Step in 7 hours.
  2. Work somewhere where everyone realizes we’re all fighting the same fight. I’m surrounded by coders and when we needed coding, 1492 python coders lined up to help.  Not a single one of them reports to me.
  3. Get upset, yell, demand results.  But realize when it’s the right time to yell and when it’s not.  During a firefight, I need you to be on the best fucking game of your entire life.  It is not the time to be berating you.  It’s the time to treat you like a hero, a magician.  It’s when I do what you tell me to do for you.
  4. Communicate the heck out of everything.  Throughout this outage we found other tools to use to let users know what was going on and what to expect.  I’d post updates even when the information I had was incomplete.  I’d say so.  I hated having folks in the dark.  
  5. Expect criticism.  Some of it will be searing.
  6. Realize that the people working under me on this are collectively smarter than I am.  Offer help whenever but let them work.  Take point at handling communication.  Make sure #5 doesn’t get to them. Remind yourself of #3.

It took nearly two days to get things back to an okay state, a state where we had new emails.  Still recovering data from backups and reconstructing state from a now corrupt MySQL database.  

I’ll probably never be able to express my gratitude to the team I manage for their efforts this week.  Sucks we got here but without thinking, I’d go to battle with this team again.

We made mistakes that got us here but we can talk about that later and make sure it doesn’t happen again.  


I’ve lived most of my life with very few friends.  Growing up most would say I was a shy introvert and while I did have childhood friends in school, outside of Facebook, I don’t really keep in touch with any of them now.  For whatever reasons, those friendships didn’t seem to stick.

My circle or friends is largely comprised of co-workers but co-workers who I know outside of work, who I talk to outside of work just because and who call/txt me for non-work reasons – just “because”. 

The one exception is a small group of friends I met while in college. 

As things turned out, my neighbors in my freshman year at college all went to the same high school and I got to know them and their circle of friends really well.  We’ve gone to each others graduations, weddings, and send holiday cards.  Time’s moved on and we’ve all drifted around the county but somewhere in the middle there the Internet happened and we’ve all kept in touch largely through emails. 

This is the group that knew the young me, the group that helped me through breakups and supported me when I decided to move to California.

We rarely see each other in person.  Tonight was special.  They are all in town for their 20th high school anniversary and I was able to make a trip out too and tonight we all got together as a small group. 


It’s a nice reminder of those who have shaped my own life.  Those who it takes just minutes to catch up with, nearly as if no time has passed since the last time you saw each other. 

It was also nice to laugh, to laugh like I haven’t laughed in a long time, about stupid things.  Stupid things we did in college or during the summers between college.  About things that have happened since college. 

We all have our own different lives now but it was truly a treat to spend a couple hours with my friends.


(This post is pretty delayed.  Had hoped to post this shortly after I got back but life happened and here I am a month later.)

I just got back from a little more than a week in Paris. I went mostly for work but since I’ve never been to France and only to Europe three times before, I tried to mix in some personal time.

One of my former colleagues whom I admire often blogs to write down his own experiences. I think that’s a neat idea and I find myself taking random notes on my phone while traveling now.  This post is a result of those notes.

I’ve never been to New York City but I imagine it to be much like Paris. A city that seemingly doesn’t have a bed time. I never got a hang of the time shift from California and would often be awake at 2am. So was the street outside the office. And so were all the restaurants around the office. The energy at 2am was as electric as the energy at 4am.

Walking around the corner at 2am to grab a crepe did not seem out of the norm. Mostly, I got to see things I had only seen in Ratatouille.

Spent a lot of time in Mozilla’s Paris office upgrading some of the network equipment and trying to understand what it means to work remote from Mountain View.

The time shift off California was something else. I’m surprised anyone at Mozilla Paris (or Europe in general) is able to function with Mountain View. 9 hours made real time dialog really difficult and really emphasized the need to record various meetings for time-shifted viewing.

I took a train down to Nice and met Cedric, one of the localizers. Nice felt like Santa Barbara in a lot of ways but with warmer water.

And here’s where Buenos Aires left an indelible mark on me. After my trip to Buenos Aires, I picked up an interest in a number of groups, including Gotán Project. They were playing at Les Nuits du Sud in Vence, just outside of Nice.

This was one of my trip’s highlights, especially when they played Santa Maria. I emailed a friend right after the show and commented to her,

Gotán Project was better than I imagined live. Such a great show. I mean I’m in France standing in a town square listening to Spanish music sung by an amazing woman with half the crowd dancing Argentine Tango. Everything I like about classical Spanish guitar & electronic music.

The most interesting thing about this trip was the number of non-Paris based Mozilla folk that kept arriving in Paris.  The first week I was there David Ascher was in town.  The following week, the US-based Jetpack team was in town. 

Paris certainly ranks in the top handful of favorite cities but still has a bit of a way to go before it can oust San Francisco.

i lost my passport in hungary

I’m posting this in the hopes others who find themselves in this predicament will find this online and find it useful. I certainly found comfort in this guy’s tale.

I took a side trip to Hungary this past weekend and inadvertently lost my passport.

So now what?

This all happened on a Friday night. Thank God for a smart phone. And data roaming be damned. I quickly googled for the embassy in Budapest and called the emergency after hours number. After declaring myself an American citizen, my call was escalated to the oncall duty manager.

Here’s where definitions of emergency differ. I has a flight out of Budapest Sunday morning. This was a “blocker” for me. The US Embassy in Budapest is closed over the weekends (even for emergencies). Not an emergency to them.

The best the duty officer could offer me was to arrive Monday morning, identify myself as an American and I’d be escorted in to get a temporary passport.

I tweet’d looking for help. I crowd sourced getting help. You have no idea how helpful that alone was. (thanks everyone!) I had people sending me DMs and text messages and replies to my tweet. It felt good to know I had this network of people willing to help me.

Saturday I retraced my steps (no luck) and moved my return flight to the last possible one on Monday. It was mentally really hard to have any fun the rest of Saturday.

Sunday. Budapest is hot, hot like Yucatán hot. Budapest is also a very walkable city. However, because of the heat I spent most of of my time going from one free wifi coffee shop to another. I also scouted out my embassy.

Monday morning. Embassy opens at 9am. I’m up at 7a, dressed and fed and out by 7:45a. Way ahead of schedule. I got to the embassy at around 8:15a and told the guard I had lost my passport and showed him my California drivers license. He disappeared for a bit and then opened the gate for me. He told me to leave my bag with him, it’ll be easier to get in (made sense to me – I wanted as little drama as possible).

Currently, my most valuable possession.

Currently, my most valuable possession.

I had to empty my pockets and literally turn off my phone before going through the metal detector. The guard there put all my belongings except for my ID and money into a box.

I grabbed a ticket and waited a few minutes until hey called my number. I had to fill out a passport application & lost passport form, get passport photos ($5.41), pay $135 for a new passport and wait 20 minutes for them to print out my passport.

At least I had alternate ID. The gentleman behind me had nothing but a copy of his passport which didn’t seem to be of any use. I talked to him a bit. Same thing, lost his passport and everything else he had in his “pouch”. At least my stuff is all separate.

Total time in embassy, 1:19. Didn’t have the patience for the metro and took a taxi to the airport.

So basically a huge inconvenience. Meant having to change a number of flights around (not free). Meant staying longer than I had packed for or planned to in Budapest.

Also, screwed my work schedule. Also, expensive mistake.

Souvenier from Budapest.

Souvenier from Budapest.

So lesson learned. Don’t lose your passport. But if you do, lose it during the week and not the Friday before the weekend.

If you’ll indulge me,

Single points of failure suck. I can’t help thinking that this whole passport concept is a single point of failure. I lost it and was screwed. Never mind that I had a couple credit cards and a California drivers license with me -and- a color copy of my passport.

I also have a this biometric data that’s physically attached to my body and REALLY hard to lose. I felt like I had all these tools to conclusively prove who I am and some computer could verify I was okay to fly.

Back in Paris

I learned long ago that home is wherever my stuff is. My stuff – laptop, luggage – was in Paris. I’m back in Paris, still far from my home but I can’t tell you how much this feels like home!

behind my name.

This came up the other day and it occurred to me that not everyone knows why I call myself what I call myself.

  1. Why mrz?
    It’s not short for Mr. Z.Shortly after moving out to Mountain View in 1996, I worked at 3Com. I worked in the engineering division that, during the two years I worked there, went by names such as NSD and ESD. This was the division that made “brouters” and if memory serves, largely came from Bridge Communications. I started there as a Solaris syadmin and left as a network engineer (and didn’t really look back).

    Anyways, username convention was your first, middle and last initial. mrz stuck. Also, it’s half as long as my first initial and last name.

    (Bonus points if anyone knows my middle name without using Google.)

  2. Matt or matthew?
    In high school I worked at Dairy Queen. One of the highlights, of course, was taking home soft serve ice cream (“mistakes”). But that’s not what this is about.When I started, my name tag read:

    Welcome Matt

    It was at this point I decided I would only go by matthew – I am neither a doormat nor a welcome mat.

    Nowadays, I answer to both, but often correct to the preferred. Which you use tends to indicate how well you know me.

  3. But why matthew and not with a capital M?
    You’ll very rarely see me write my name, first or last, with any capitalization. This an artifact of my first email address (, which was in all lower case.That stuck. So did the fixed-width font. It’s weird, I know.
  4. One more thing…
    Since I already have you at 3 bullet points, here’s one extra bit of trivia.My first name comes from my great grandfather’s middle name and this first century Galilean. My middle name comes from my grandfather’s first.

buenos aires

Last week, for eight days, I was able to step outside my normal role managing Operations and wear an entirely different hat.

I had an amazing opportunity to interact with the vibrant Mozilla and Open Source community; I got to interact with those who I help from the shadows every day.

I’ve had a couple days to let my thoughts soak in.

Something about Argentina and Buenos Aires resonated with me in a way that’s hard to describe. It is, perhaps, the first time in my life I’ve had a sense of reverse home sickness. From San Telmo to La Boca Caminito to Palermo to Recoleta Cemetary and Calle Florida, Buenos Aires oozed of culture. From pizza to empanadas to more gelato than I can remember, it’s unlike anywhere else I’ve ever traveled.

Persicco, best gelato around.

Persicco, best gelato around.

The bustling energy of San Telmo’s Feria de Antigüedades was matched only by the energy of the open source community I met.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

On Wednesday night I had the pleasure of attending the Firefox 4 Party. This was amazing. The night before I met with several of the event organizers, including Guillermo Movia. They were expecting 50 or so and instead had 150 at the party. I don’t know how to really describe what it was like, walking around and mingling with everyone (in my broken Spanish no less), hearing everyone talk about Firefox and Mozilla.

Thursday night we attended the first Hacks/Hackers MeetUp in Buenos Aires at AreaTres. The discussion was all in Spanish but I mostly kept up. I was amazed at the turn out. Was a far larger group than I would have imagined and made me realize how large the open source community in Buenos Aires is.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

Friday wrapped up with a Design MeetUp at Urban Station that Tara led. This turned out to be one of the surprise highlights mostly because of the discussion afterwards. I don’t often get to interact with the community in such an intimate venue and speak Mozilla.

You can take me out of networking but you can’t take networking out of me. At each place we went to I’d always check to see who I had upstream connectivity from and what my path to Phoenix or San Jose looked like. Urban Station had the quickest Internet I had experienced while in Buenos Aires.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

On a personal side, since all of these events were after 6pm local time and I was shifted 4 hours off California, I found a lot of time to explore and soak in Buenos Aires. I walked more than I can remember, slowed down more than usual to look and listen. Ate. Indulged. Walked. Explored. Saw a ballet show at Teatro Colón. Went to a Tango show. Walked to Carlos Gardel’s house (Casa Museo Carlos Gardel) in Abasto. Inadvertently walked to Palermo and had mint iced tea. Bought a crappy umbrella and walked in the rain.

No doubt I was lucky to have a fantastic travel companion (I’ve thanked you, haven’t I Tara?).

Lastly, I want to share this:

I went with very little expectations and maybe a little nervous anticipation. I came back with a profound sense of Mozilla, of the community that supports Mozilla and a feeling of renewed purpose for why I work at Mozilla. I came back with more friends than I left with, with a twitter feed half in Spanish.

John Lilly used to talk about about great companies vs good companies. How great companies last; they may change but their mission remains. Mozilla, he argued, was on a path to be a great company. Today, the vehicle for Mozilla’s mission is Firefox. Tomorrow it could be something else. But the mission will remain.

This is the sense of Mozilla I was left with when I landed in San Francisco. The emotional connection people make with Mozilla, and more precisely, its Mission, is what will make Mozilla one of the great companies.

I’ll leave you with a couple pictures I took. Tara did a better job taking photos than I did – you should check out her Flickr gallery.

Community Events

Mozilla, Firefox, “fixing computers” & a room full of kindergartners

My son has a vague concept of what I do at work. He knows I work at “Firefox”, knows the dino and knows I “fix computers”.

He knows that if he wants to get online he has to double-click on the Firefox icon and then on the Kidzui icon:

kidzui-toolbarHe asked me if I could come to his school and show people how I “fix computers”.

I started off by asking if anyone recognized the logo on the back of my shirt.  Even to a room full of 5 year olds, the Firefox logo was instantly recognizable – there wasn’t anyone who didn’t know what it was.  I talked briefly about what Mozilla did (“we make a web browser”) and that I help fix computers when they break.

I thought about showing them Mitchell’s Mozilla Tree but probably couldn’t have done as well as Mitchell could have!

Then I pulled out my laptop and showed them Firefox (Minefield really).  I was in the middle of showing them Firefox and how what my son does when he wants to get on the Internet… and Minefield crashed.  Which was a great segue into “lets go look at these computers I brought and how I fix them!”

I brought three old Celeron “servers” (you can hardly call a Celeron a server) with lids off and we spent the next 20 minutes taking apart the machines.  We took out the computer’s brain and the fan to keep it cool (they didn’t believe me that you could cook food on the CPU when the computer was “thinking hard”).  We took out the two memory sticks and the hard drive and the IDE cable.

These are things most parents in their right mind wouldn’t do with their home computer and these kids really enjoyed physically touching these parts and asking questions.

Mary Colvig helped me gather up a bunch of Firefox bags and my two kids and I had stuffed stickers into each one. Kids went crazy over the Foxkeh stickers and there was a collective “awwww!” when I showed them the “don’t hurt the web” stickers!

Anyways, good times. I enjoyed talking about Mozilla in a very different setting than I’m used to.

I’m going to go work on my new role in community outreach now…

my face is cold

NSID 2008 had to come to an early close for me today. I leave Monday for my sister’s wedding in Chicagoland and while I would have loved to hold out till the end of the month or even a few more days, that just wasn’t realistic (something about looking presentable).

You could argue I could have waited a couple more days but shaving at this scale is best done at home and not in a hotel!  Too bad too because it was just starting to get comfortable.

Here’s to next year!

doing as my father did

I have some pretty fond memories of my Dad, a math teacher, bringing home one of the (or maybe it was the) Apple II computers from work when I was in first or second grade.  That’s how I got introduced to computers (and where I learned how to spell catalog).  It’s also where I spent a lot of time learning economics and running my own lemonade stand (iPhone version here).

computertimeFlash forward 30 some years and I find I’m doing the same with my two children and weirdly enough on Apple computers (first computer I owned was  Commdore 64 followed by an Amiga and then some Gateway running Windows in 1995 – only recently went back to Apple).  My son spends his time in KidZui (which to him is the same as and my daugther does drawing with Tux Paint (on an 8 year-old Powerbook, but she doesn’t seem to notice or care).

My son recently asked me if he could have my black MacBook when he is older.  I laughed, sure, but I haven’t any idea what computers will look like 30 years from now!

My Kids & the California Academy of Sciences

CA Academy of Sciences, Living Roof

CA Academy of Sciences, Living Roof

This is far from my normal Mozilla related posts but worthwhile enough that I feel like sharing it.

I took my two children to the California Academy of Sciences this weekend.  I wasn’t sure what to expect but since it’s new (or newly opened) and I enjoy these types out outings with my children, this become our weekend activity.

It’s a combination aquarium (which saves me a trip to Monterey), natural history museum, planetarium, and rain forest.  The rain forest is enclosed in a 4-story glass-like globe with a winding staircase that, I think, lead to the roof top. The line was too long to bother going in with two children who clearly wanted to run around and explore instead.  There was also a great outdoor area on the west side of the building where the kids had fun running around while having snacks.

One of the places I’ve missed from Chicago was the Field Museum. The natural history part wasn’t anything as large as Chicago’s but was good enough to make me want to go back.

The absolute best, however, was when at bed time when my five-year old son told me that today was the best day ever.  So thanks, Academy.  You’ll be getting my membership application soon.

Things learned:

  1. Buy tickets beforehand.  The lines were really long and the Planetarium was sold out by the time we got there.
  2. Let the kids loose and let them tell you where to go and what to look at.  Turns out they know how to explore best.
  3. Get a membership.
  4. Museum has guest WiFi.