i think

[This is a little old, probably written sometime in 1993.]


I think of the stars in the sky
   how lonely they must be,
   so far apart from each other.

I think of a candle without a flame,
   missing that which makes it whole.

I think of the winter trees with their leaves all gone,
  how sad they must feel.

I think about the tumbleweed,
  blowing across the open land.
  Lost, wandering about, not knowing where to go.

And then I think about you
  and suddenly,

        the stars aren’t so lonely,
        the trees aren’t so sad,
        the candle shines brightly,
        and the tumbleweed…
        the tumbleweed isn’t so lost.

one planet. one people.

I wound up taking most of the day off from work yesterday to drive 200 miles to help a Friend.  It was a long and exhausting day.  This is a person who ten months ago I didn’t even know, let alone care about.

But this isn’t a post on what I did.

It’s that I was struck by a number of people and things that came together to help my Friend.  A relatively large number of selfless people.  Enough that thinking about it makes my eyes water.  Enough that even writing this post causes me to choke up.

I was struck that amongst all those who I interacted with to help my Friend, not a single person did so with any expectations or desires of anything in return.  We helped, simply, because it is what you do.

  • There was the person at work who made sure I was going to drive my Friend.  She made it clear that that not helping would have been a personal disappointment.
  • There was the person who moved mountains in what felt like record time to make sure my Friend had certain resources.
  • There was my other friend who did me a favor to help my Friend, who pulled in his own circle to help.

Lastly, my children.  My children have meet my Friend exactly twice and once was last night.  Yet they exhibit all the passion and care for humankind a parent could only wish for.

  • There was my daughter who made it absolutely clear that I needed to make sure my Friend knew to call me whenever, should my Friend need help.  My seven year old daughter.
  • There was my son who checks in with me daily asking how my Friend is doing, who reminds me we have a spare room should my Friend need a place to stay. My nine year old son.

There is that adage that it takes a village to raise a child.  The truth in that felt real yesterday.  There was this sense all day that here we all are, many voices, one planet, one people.

39.

I turned 39 this past Sunday. 

My daughter and I share the same birthday (as does @selenagomez!). Over the years this has turned into a fun game of arguing over whose birthday it is.  And over these years my priorities of what I want on my birthday have changed.

  1. Farrell’s Ice Cream Parlor.

    When I was younger I remember going here for my birthday.  All the pomp and circumstance! This place more or less disappeared.  When I saw they had re-opened a few in Southern California I knew what we were going on my LIZ’s birthday.

    I couldn’t have been happier when the band came out, drums and horns blaring and LIZ on her chair. She’s never seen anything like this. She had this crazy big smile on her face as she soaked up being the center of attention (for literally the entire restaurant).

    In that very instant I saw the young woman she will eventually become. Full of confidence and poise.

  2. My son took me to lunch.

    Those who know me know I enjoy a great culinary experience. I like tasting new foods. I love the presentation.

    When WNZ asked me what I wanted for my birthday, I told him I only wanted him to take me out for lunch (well all four of us). I figured I’d give him some cash and let him treat. The more we talked about it the more I could tell he was getting excited about doing it.

    He took me out to lunch at Opah Restaurant, a place my wife and I used to visit when we lived in the area. I kept trying to give him money but he’d have none of that.

    “Do you need some money?”

    “No Dad, I’ve got this.”

    And in that instant, I saw the young man that he will eventually become. Full of independence and pride.

I may not be a man with a lot of material wealth but nothing could have made me happier than those two things this weekend.

Zimbra & Mozilla email, 4 months later

Four months ago was a very tough time in Operations. We suffered a catastrophic disk array failure on Mozilla’s mail server (I blogged about it too). A series of mistakes kept email offline for two days. This was the worst I’ve ever felt, both professionally and personally.

Fast forward to today. So. Much. Better.

We learned. We researched. We re-organized ourselves. Much like The Six Million Dollar Man, we rebuilt it better.

justdave posted his account, “Re-imagining Zimbra email at Mozilla” but I wanted to add my own color.

Background

During my interview at Mozilla in 2006, I was asked a bunch of questions about Zimbra. First I had heard of it. By the time I started I had learned quite a bit about Zimbra. Back in 2006, mozilla.com email was hosted externally and we began the process of moving email back in house. The company hosting email couldn’t provide SSL and wasn’t doing all the groupware things we needed.

Post Zimbra-gate (December)

I was mentally done with email. I looked at simply outsourcing. I looked at hosted Zimbra, hosted Exchange, hosted whatever.

We Mozillians, we’re a unique group.

  • We want to use the IMAP client of our choice. Some of us just want to use the web interface. Others prefer Microsoft Outlook. Or Thunderbird, or Mail.app or Postbox or mutt or pine or Sparrow or …
  • Calendaring is just as complex.
  • We need to support a wide number of mobile devices – iOS, Android, Blackberry, devices that support Microsoft’s ActiveSync – with both email and calendaring.
  • Some use Zimbra’s document sharing/storage
  • We need something that supports IMAP, ActiveSync, CalDAV, CardDAV.

We looked at what others at our scale and beyond our scale use for email. Oracle uses Zimbra. Comcast uses Zimbra. At. Scale.

We talked to others hosting their corporate email with Google Apps (and their 15-person staff managing their Google Apps mail!). We learned that deploying Exchange requires a move from OpenLDAP to Active Directory and a particular skill set that we don’t have in house.

Moving Forward

This incident highlighted the need to have a team focused on infrastructure. Our primary focus (and priorities) always tend to lean towards various Mozilla web properties or developer services.

So we did two things –

  1. Broke up a fairly flat Operations group and created an Infrastructure Operations team (and a couple others) to focus on services like email & LDAP, to name a few.
  2. Built a new environment for services that, when break, cause work stoppage, cause a line to form behind my desk. This Hyper Critical Infrastructure, or HCI, is isolated from the rest of the production environment, has different change control processes and is meant to hit as many “9s” as we can hit. It’s a very different way of planning than we had done in the past.This technology stack uses more corporate/enterprise technology than we’re used to using at Mozilla.

HCI Today

HCI straddles two high density, (~15kW) racks. It’s only relation to the rest of Mozilla production network is two 10GbE fiber drops from the network core.

HCI has it’s own Juniper SRX 1440 firewalls. Its own Juniper EX4500 switching. Its own NetApp FAS3270. Its own 5 node VMware ESX cluster, each machine having 2x 6-core Xeons & 192GB RAM.

In a couple months, services here will be replicated to SCL3 using various NetApp & VMware technologies.

We had planned to have HCI in production by the end of February but no one wanted to rush this (plus someone decided to have a baby).

Instead we slipped that to the last week of March and I’m glad we did. We consulted with Zimbra and others. We sent Desktop & InfraOps to training. We tuned and fine tuned.

Zimbra Today

We have mailboxes spread across seven mailbox servers and understand the metrics we’ll use to determine when to add more mailbox servers.

We migrated 1002 mailboxes from San Jose to Phoenix without anyone noticing, without any user impact, in just a couple days. In fact, we didn’t mention it until we were done.

We have instrumentation and trending and alerting on everything we could think of.

What’s next?

All is for naught without learning. We learned a lot and we’ve changed how we operate as a team.

Once bitten, twice shy.

A Test

[This is a short story written in 1991 while at UIUC.  Lyric quotes are from some Enya song, name escapes me now.]


He stepped out into the cold, glad to be through and glad to be out.  The cold winter wind slapped him in the face as if scolding.  He hadn’t really bundled up too well – just wanted to leave.  He stood there now pulling together his jacket, fastening the zipper and throwing his bag over his shoulder.  Sure was cold tonight.

“Damn…” he though to himself.  “What the hell’s happening to me?”  The cold wind stopped and stood still and for a brief moment, there was silence around him.  He reached into his bag, felt around, and produced his walkman, perhaps the only thing he valued tonight.  “Got a long walk back.”

Pretty cold tonight.  Pretty dark too, and quiet, as if everyone knew and wanted to leave him alone.  Everyone else was probably still in there.  There were perhaps many going on tonight and many people were probably still finishing up.  He didn’t care – just glad to be out of there.  “No use in sitting around here any more.” he has said to himself as he had grabbed his bag to leave.

It was cold.  Cold and dark.  The wind had picked up.  He had felt alone before, but tonight he felt really alone – and sad.  He had never failed like this before.  “Fuck it – who gives a shit anyways?” he said out loud, breaking the stillness, as he flicked on his walkman.  As if to abuse him, a brief gust of wind blew by.  His ears stung.  He could imagine what they would say, and wanted only to block it out.  He shrugged.  “Life goes on.”  He shrugged again, as if to signify something important, and started back.  He shoved it back into his pocket.

Finding nothing interesting on the radio, he hesitantly freed his hand to brave the cold, and thumbed for the play button.  For the past five months, music had become his one place of refuge.  He could block out anything with the flick of a switch.  If only that worked in real life. ‘…Eurus…’  He still had a ways to go.

It was cold and reminded him of happier days; days gone by delivering papers.  How he was alone and could be himself.  How he would daydream of his other life (how he wished he was there instead of here), and for a short while, all would be okay.  How he would yearn for the house up next with the Christmas lights strung up, smiling at him.  He would pay them a little attention as he stopped to warm his hands in their glow.  ’…if every man is true…’

“How romantic a night!” he thought.  “How I wish I had someone special in my life.” A though which brought a tear to his eye.  Someone to take a walk with on a night like this.  Someone who would care because tonight he didn’t.  He’d never been in love before.  He rounded the corner and could see the lights ahead.

This wasn’t his life, was it?  He wished with all his heart that someone would come and take him away.  Away to the place he belonged.  But that was there.  ‘…etu itu ad astrum…’  He was here.  The wind stopped yet again, and there was the noise of passing cars and people talking.

How warm he felt as he pushed open the door to the builing and headed down the hall.  She saw him quickly, and started towards him.

“Your back early!”

“Yeah, it was a breeze.”

recovering from an email outage

If I could do this week over I would.  Too bad I can’t.

Email today is vital.  Not having it makes your heart palpitate. 

Monday morning, during a swap of a failed hard drive (something we’ve done countless times) the storage array we use for email went offline.  The whole thing.  And for various reasons, the last known good backup was from awhile ago. 

I painfully remember thinking “oh shit” when I realized what this meant.

[This isn’t a post about all the things I should have done to make sure I was never in this spot.  Everything’s obvious now.]

I learned a couple things this week:

  1. Hire the absolute best people (and geezus, hire people smarter than you!). You never know when you’ll need them.  You never know who will have the answer to the problem.  Hire people who care about each other.  You never know when you need them to look out for the one guy who, in 73 hours, forgot to sleep.  The same one guy who has to run point on The Next Big Step in 7 hours.
  2. Work somewhere where everyone realizes we’re all fighting the same fight. I’m surrounded by coders and when we needed coding, 1492 python coders lined up to help.  Not a single one of them reports to me.
  3. Get upset, yell, demand results.  But realize when it’s the right time to yell and when it’s not.  During a firefight, I need you to be on the best fucking game of your entire life.  It is not the time to be berating you.  It’s the time to treat you like a hero, a magician.  It’s when I do what you tell me to do for you.
  4. Communicate the heck out of everything.  Throughout this outage we found other tools to use to let users know what was going on and what to expect.  I’d post updates even when the information I had was incomplete.  I’d say so.  I hated having folks in the dark.  
  5. Expect criticism.  Some of it will be searing.
  6. Realize that the people working under me on this are collectively smarter than I am.  Offer help whenever but let them work.  Take point at handling communication.  Make sure #5 doesn’t get to them. Remind yourself of #3.

It took nearly two days to get things back to an okay state, a state where we had new emails.  Still recovering data from backups and reconstructing state from a now corrupt MySQL database.  

I’ll probably never be able to express my gratitude to the team I manage for their efforts this week.  Sucks we got here but without thinking, I’d go to battle with this team again.

We made mistakes that got us here but we can talk about that later and make sure it doesn’t happen again.  

reconnecting

I’ve lived most of my life with very few friends.  Growing up most would say I was a shy introvert and while I did have childhood friends in school, outside of Facebook, I don’t really keep in touch with any of them now.  For whatever reasons, those friendships didn’t seem to stick.

My circle or friends is largely comprised of co-workers but co-workers who I know outside of work, who I talk to outside of work just because and who call/txt me for non-work reasons – just “because”. 

The one exception is a small group of friends I met while in college. 

As things turned out, my neighbors in my freshman year at college all went to the same high school and I got to know them and their circle of friends really well.  We’ve gone to each others graduations, weddings, and send holiday cards.  Time’s moved on and we’ve all drifted around the county but somewhere in the middle there the Internet happened and we’ve all kept in touch largely through emails. 

This is the group that knew the young me, the group that helped me through breakups and supported me when I decided to move to California.

We rarely see each other in person.  Tonight was special.  They are all in town for their 20th high school anniversary and I was able to make a trip out too and tonight we all got together as a small group. 

Amazing.

It’s a nice reminder of those who have shaped my own life.  Those who it takes just minutes to catch up with, nearly as if no time has passed since the last time you saw each other. 

It was also nice to laugh, to laugh like I haven’t laughed in a long time, about stupid things.  Stupid things we did in college or during the summers between college.  About things that have happened since college. 

We all have our own different lives now but it was truly a treat to spend a couple hours with my friends.