Rebooting Community IT, Part 3

[After Part 1 & Part 2 I thought I was done. But the more I reflected on what Community IT could be I realized there’s a Part 3 where I’ll share some of the thinking we came up with that resulted in a Draft Proposal.]

The CSA

This past January I started working with my former colleague @thunder (currently at Creative Commons) and my friend and then current colleague @solarce (Mozilla Webops) on what it would be like to have an organization comprised of volunteer SysAdmins who would lend their time to help manage infrastructure & systems for open source and public benefit organizations. Not too unlike the Open Source Lab but extending to not just open source projects but organizations like Creative Commons or Mozilla.

We called this Community SysAdmins or CSA.

Some of the thinking that went into the January draft was based on our collective experiences at Mozilla, especially around organizational governance – modules and module owners, for example.

My thoughts around this continued and I began to think more about reasons why Community IT within Mozilla hadn’t taken off. Or why it had taken off very slowly.

I started to see a model where it could work and it was not unlike the Mozilla Reps model.

ReMo structure (draft)

ReMo structure (draft)

  • CSA Advisory Board Leadership Council (~ ReMo council)
  • Core group of Community SysAdmin Mentors (~ Mozilla Reps)
  • Community IT/SysAdmin/Ops volunteers (~ Mozilla contributors)

Rebooting Community IT with CSA

As we start thinking about what it means to Reboot Community IT, I want to share the draft CSA proposal we created and re-ask these modified five questions I asked in Part 2:

  1. What impact does this group want to have on open source projects and public benefit organizations?
  2. How do we become a force multiplier for your organization?
  3. What are our 9-12 month goals and objectives?
  4. How do we do it?
  5. How do we work with your existing IT Operations as partners?

And ask one more:

  1. How do we create the CSA to support organizations like Mozilla or Creative Commons, for example?

Continue reading

Rebooting Community IT, Part 2

[This is Part 2. In Part 1 I wrote about where we started from and where we are today. In this post I’ll share where I want us to go.]

From 2011 to 2013, Mozilla IT’s impact on the Mozilla Community lead to Air Mozilla and Community Website/Email Hosting.

CSAimageThe holy grail was something more. Something where volunteer contributors were running or helping run websites and working with Mozilla IT Operations.

Something where there is a “talented group of volunteer SysAdmins who want to devote some of their time to help the open source and open communities with non-profit organizations that work to create open source and a better, open Web.”

[I blatantly lifted that off communitysysadmins.org – an effort I started with a small cohort of people early in 2013.]

Mozilla & Community IT Operations
There hasn’t ever been a Community IT Operations anything at Mozilla.

Throughout my tenure at Mozilla we’d occasionally spin up servers or services that would be managed by someone in the community. We’d even support whole projects that were related to Mozilla but largely independent projects (Camino & SeaMonkey, for instance).

But unfortunately this felt very ad-hoc. These where difficult to support within Mozilla IT Operations.

What’s Next?
With my Mozilla contributor hat on, I want to build a new movement.

I want to start with a focus on Mozilla and build a Community IT / SysAdmin / Operations movement.

And I know we can do it.

We will be fierce. No one will build into the Internet the kinds of things we want to build.

Community IT Operations Kick Off Meeting
I want to invite you to join my movement and our kick off meeting. We’re still sorting out then when (scheduling is hard – check out this Doodle to help me find a time).

In the mean time, think about how you’d answer these questions:

  • What impact does this group want to have on the Mozilla project?
  • How do we become a force multiplier for Mozilla?
  • What are our 9-12 month goals and objectives?
  • How do we do it?
  • How do we work with Mozilla IT Operations as partners?

Join communityit@mozilla.org and watch https://wiki.mozilla.org/IT/Community for updates.

Join me?

We will be fierce. No one will build into the Internet the kinds of things we want to build.

Rebooting Community IT, Part 1

It’s been awhile since I talked about (Mozilla) Community IT.

In Part 1 I want to recap where we started from and where we are today. In Part 2 I’ll talk about What’s Next.

In October 2011 I asked the following question:

“How can Mozilla IT impact the Mozilla Community?”

Within Mozilla IT I helped focus our efforts around three pillars:

  1. Community IT
  2. Community/Volunteer Sysadmins
  3. Air Mozilla

I was even so bold to call out three specific goals for the end of 2012:

  1. to have 5-10 volunteer Community Sysadmins actively helping run Mozilla’s network and servers.
  2. to have a vibrant Community IT group…
  3. to have a premiere source for open source video technology, a site where the Mozilla Community can find, share and create video content

I feel that I generally failed. Or rather, I did not meet the goals I set out.

But I’ve talked about failure before and the Mozilla IT I left is not the same that it was in 2011. In fact, on behalf of my former team, I’ll claim the following two successes:

  1. Air Mozilla.

    In 2011 this wasn’t a thing. No one cared about it.

    Nearly a year later we were discussing specific policies on what we’d stream (public? internal?). A year later it was assumed everything we did would be streamed.

    Air Mozilla became one of the primary communication tools at Mozilla.

  2. Created infrastructure for Community Website/Email Hosting.

    Through Mozilla Reps, Mozilla IT worked to provide Email hosting for 13 communities and Website hosting for more than 29 hosted sites.

Essentially, Mozilla IT literally empowered the Mozilla Community to promote the Mozilla Mission.

Ironically, that was the over arching goal I called out in the third paragraph October 2011.

So maybe I didn’t fail as much as I thought?

on leaving mozilla

Mozilla IT 2012

Mozilla IT 2012

Last Wednesday, July 31, I left Mozilla as a full time paid-staff.

Seven years at Mozilla has fundamentally changed how I operate. It’s redefined my values and helped define my personal mission & purpose.

The impact I had on Mozilla and on the Mozilla Project extended far outside Mozilla IT Operations and I hope to continue to be connected to the Mozilla Project (but more on that in another post).

For now I’ll simply share the note I sent to Everyone@Mozilla:

Adios, au revoir, sayonara

On March 15, 2006 I quietly joined Mozilla. Seven years and four months later, I quietly take my leave.  July 31 will be my last day here.

I had this moment of clarity awhile ago when I realized I have an awesome set of leaders under me and it was time, as Mitchell has said, to let them step into new roles and for me to step out.

To my Mozilla IT:

I have enjoyed being there for you.

I have enjoyed seeing you grow.

I have enjoyed seeing you handle some crazy, complicated challenges.

I will miss the camaraderie we developed and shared with each other.

I will miss the late nights working with you.

You always worked with me, never for me. And as much as I tried to help you, you helped me.

For this, and much much more, I thank you.

To my friends,

I made friends with people I worked with, people who are friends first and co-workers second.  I will miss seeing you every day. This is probably the hardest realization.

You have helped me grow and become who I am today.

To All@Mozilla,

In many ways, you, and this organization, have made an impact on me that cannot be undone.

My thoughts and actions have been shaped by what we have collectively been accomplishing and what we have set out to accomplish.

I could write much much more, of course so I’ll just leave with this:

“Parting is such sweet sorrow that I shall say goodnight till it be morrow.”

Zimbra & Mozilla email, 4 months later

Four months ago was a very tough time in Operations. We suffered a catastrophic disk array failure on Mozilla’s mail server (I blogged about it too). A series of mistakes kept email offline for two days. This was the worst I’ve ever felt, both professionally and personally.

Fast forward to today. So. Much. Better.

We learned. We researched. We re-organized ourselves. Much like The Six Million Dollar Man, we rebuilt it better.

justdave posted his account, “Re-imagining Zimbra email at Mozilla” but I wanted to add my own color.

Background

During my interview at Mozilla in 2006, I was asked a bunch of questions about Zimbra. First I had heard of it. By the time I started I had learned quite a bit about Zimbra. Back in 2006, mozilla.com email was hosted externally and we began the process of moving email back in house. The company hosting email couldn’t provide SSL and wasn’t doing all the groupware things we needed.

Post Zimbra-gate (December)

I was mentally done with email. I looked at simply outsourcing. I looked at hosted Zimbra, hosted Exchange, hosted whatever.

We Mozillians, we’re a unique group.

  • We want to use the IMAP client of our choice. Some of us just want to use the web interface. Others prefer Microsoft Outlook. Or Thunderbird, or Mail.app or Postbox or mutt or pine or Sparrow or …
  • Calendaring is just as complex.
  • We need to support a wide number of mobile devices – iOS, Android, Blackberry, devices that support Microsoft’s ActiveSync – with both email and calendaring.
  • Some use Zimbra’s document sharing/storage
  • We need something that supports IMAP, ActiveSync, CalDAV, CardDAV.

We looked at what others at our scale and beyond our scale use for email. Oracle uses Zimbra. Comcast uses Zimbra. At. Scale.

We talked to others hosting their corporate email with Google Apps (and their 15-person staff managing their Google Apps mail!). We learned that deploying Exchange requires a move from OpenLDAP to Active Directory and a particular skill set that we don’t have in house.

Moving Forward

This incident highlighted the need to have a team focused on infrastructure. Our primary focus (and priorities) always tend to lean towards various Mozilla web properties or developer services.

So we did two things –

  1. Broke up a fairly flat Operations group and created an Infrastructure Operations team (and a couple others) to focus on services like email & LDAP, to name a few.
  2. Built a new environment for services that, when break, cause work stoppage, cause a line to form behind my desk. This Hyper Critical Infrastructure, or HCI, is isolated from the rest of the production environment, has different change control processes and is meant to hit as many “9s” as we can hit. It’s a very different way of planning than we had done in the past.This technology stack uses more corporate/enterprise technology than we’re used to using at Mozilla.

HCI Today

HCI straddles two high density, (~15kW) racks. It’s only relation to the rest of Mozilla production network is two 10GbE fiber drops from the network core.

HCI has it’s own Juniper SRX 1440 firewalls. Its own Juniper EX4500 switching. Its own NetApp FAS3270. Its own 5 node VMware ESX cluster, each machine having 2x 6-core Xeons & 192GB RAM.

In a couple months, services here will be replicated to SCL3 using various NetApp & VMware technologies.

We had planned to have HCI in production by the end of February but no one wanted to rush this (plus someone decided to have a baby).

Instead we slipped that to the last week of March and I’m glad we did. We consulted with Zimbra and others. We sent Desktop & InfraOps to training. We tuned and fine tuned.

Zimbra Today

We have mailboxes spread across seven mailbox servers and understand the metrics we’ll use to determine when to add more mailbox servers.

We migrated 1002 mailboxes from San Jose to Phoenix without anyone noticing, without any user impact, in just a couple days. In fact, we didn’t mention it until we were done.

We have instrumentation and trending and alerting on everything we could think of.

What’s next?

All is for naught without learning. We learned a lot and we’ve changed how we operate as a team.

Once bitten, twice shy.

recovering from an email outage

If I could do this week over I would.  Too bad I can’t.

Email today is vital.  Not having it makes your heart palpitate. 

Monday morning, during a swap of a failed hard drive (something we’ve done countless times) the storage array we use for email went offline.  The whole thing.  And for various reasons, the last known good backup was from awhile ago. 

I painfully remember thinking “oh shit” when I realized what this meant.

[This isn’t a post about all the things I should have done to make sure I was never in this spot.  Everything’s obvious now.]

I learned a couple things this week:

  1. Hire the absolute best people (and geezus, hire people smarter than you!). You never know when you’ll need them.  You never know who will have the answer to the problem.  Hire people who care about each other.  You never know when you need them to look out for the one guy who, in 73 hours, forgot to sleep.  The same one guy who has to run point on The Next Big Step in 7 hours.
  2. Work somewhere where everyone realizes we’re all fighting the same fight. I’m surrounded by coders and when we needed coding, 1492 python coders lined up to help.  Not a single one of them reports to me.
  3. Get upset, yell, demand results.  But realize when it’s the right time to yell and when it’s not.  During a firefight, I need you to be on the best fucking game of your entire life.  It is not the time to be berating you.  It’s the time to treat you like a hero, a magician.  It’s when I do what you tell me to do for you.
  4. Communicate the heck out of everything.  Throughout this outage we found other tools to use to let users know what was going on and what to expect.  I’d post updates even when the information I had was incomplete.  I’d say so.  I hated having folks in the dark.  
  5. Expect criticism.  Some of it will be searing.
  6. Realize that the people working under me on this are collectively smarter than I am.  Offer help whenever but let them work.  Take point at handling communication.  Make sure #5 doesn’t get to them. Remind yourself of #3.

It took nearly two days to get things back to an okay state, a state where we had new emails.  Still recovering data from backups and reconstructing state from a now corrupt MySQL database.  

I’ll probably never be able to express my gratitude to the team I manage for their efforts this week.  Sucks we got here but without thinking, I’d go to battle with this team again.

We made mistakes that got us here but we can talk about that later and make sure it doesn’t happen again.  

buenos aires

Last week, for eight days, I was able to step outside my normal role managing Operations and wear an entirely different hat.

I had an amazing opportunity to interact with the vibrant Mozilla and Open Source community; I got to interact with those who I help from the shadows every day.

I’ve had a couple days to let my thoughts soak in.

Something about Argentina and Buenos Aires resonated with me in a way that’s hard to describe. It is, perhaps, the first time in my life I’ve had a sense of reverse home sickness. From San Telmo to La Boca Caminito to Palermo to Recoleta Cemetary and Calle Florida, Buenos Aires oozed of culture. From pizza to empanadas to more gelato than I can remember, it’s unlike anywhere else I’ve ever traveled.

Persicco, best gelato around.

Persicco, best gelato around.

The bustling energy of San Telmo’s Feria de Antigüedades was matched only by the energy of the open source community I met.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

On Wednesday night I had the pleasure of attending the Firefox 4 Party. This was amazing. The night before I met with several of the event organizers, including Guillermo Movia. They were expecting 50 or so and instead had 150 at the party. I don’t know how to really describe what it was like, walking around and mingling with everyone (in my broken Spanish no less), hearing everyone talk about Firefox and Mozilla.

Thursday night we attended the first Hacks/Hackers MeetUp in Buenos Aires at AreaTres. The discussion was all in Spanish but I mostly kept up. I was amazed at the turn out. Was a far larger group than I would have imagined and made me realize how large the open source community in Buenos Aires is.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades

Friday wrapped up with a Design MeetUp at Urban Station that Tara led. This turned out to be one of the surprise highlights mostly because of the discussion afterwards. I don’t often get to interact with the community in such an intimate venue and speak Mozilla.

You can take me out of networking but you can’t take networking out of me. At each place we went to I’d always check to see who I had upstream connectivity from and what my path to Phoenix or San Jose looked like. Urban Station had the quickest Internet I had experienced while in Buenos Aires.

San Telmo’s Feria de Antigüedades

San Telmo’s Feria de Antigüedades


On a personal side, since all of these events were after 6pm local time and I was shifted 4 hours off California, I found a lot of time to explore and soak in Buenos Aires. I walked more than I can remember, slowed down more than usual to look and listen. Ate. Indulged. Walked. Explored. Saw a ballet show at Teatro Colón. Went to a Tango show. Walked to Carlos Gardel’s house (Casa Museo Carlos Gardel) in Abasto. Inadvertently walked to Palermo and had mint iced tea. Bought a crappy umbrella and walked in the rain.

No doubt I was lucky to have a fantastic travel companion (I’ve thanked you, haven’t I Tara?).

Lastly, I want to share this:

I went with very little expectations and maybe a little nervous anticipation. I came back with a profound sense of Mozilla, of the community that supports Mozilla and a feeling of renewed purpose for why I work at Mozilla. I came back with more friends than I left with, with a twitter feed half in Spanish.

John Lilly used to talk about about great companies vs good companies. How great companies last; they may change but their mission remains. Mozilla, he argued, was on a path to be a great company. Today, the vehicle for Mozilla’s mission is Firefox. Tomorrow it could be something else. But the mission will remain.

This is the sense of Mozilla I was left with when I landed in San Francisco. The emotional connection people make with Mozilla, and more precisely, its Mission, is what will make Mozilla one of the great companies.

I’ll leave you with a couple pictures I took. Tara did a better job taking photos than I did – you should check out her Flickr gallery.

Community Events