Bits about honesty

In March 2006, a large chunk of the french web went down.

There was a very simple explanation to this: Redbus Interhouse (the datacenter where most of the french web hosting providers were located) had a global power outage (link in french). Well, not just one, but two outages — and the second one was due to a human mistake.

We did not know what was happening

Someone broke the interwebz.

Someone broke the interwebz.

Most french websites were down for a few hours. The biggest blog-hosting service was entirely down (about 200,000 blogs were down, Netvibes was out, so was Vpod, etc.). It happens. Shit happens. It shouldn’t, but it does (And it was 2006, so redundancy wasn’t obvious at the time).

The worst part of all this is that no one knew what was happening. No press conference or public announcement saying “we screwed up, we’re on it”. It took about 3 days for the CEO to give a public apology (link in french). And it was an interview by an external website. Not good. Basically, the first outage was due to failing UPS (definition), and in the interview the CEO of Redbus gave a ton of details on what went wrong.

The explanation for the second outage was the following:

 A week ago, you had a similar failure?
- No, it was human error coming from one of our providers.

Take responsibility

This answer was not extensive enough. Redbus’ customers flew away in masses. If you don’t explain what happened, who screwed up and what decisions you took to avoid  a similar situation in the future, things will be bad. Trust is a very volatile concept, and one screw up can hurt you or your company  a lot if your service is critical for your customers.

A screw up can happen on a lot of different levels, be it emotional, technical, health-related… Some people lose money because of other people’s mistakes. Some can lose a lot more. Some can even die. We’ll stay on topic and only discuss a tech screw-up here.

Shit  broke here. They couln't be clearer.

Shit broke here. They couln’t be clearer.

How CloudFlare broke a part of the Internet

CloudFlare is a content delivery network (CDN) that is basically supposed to help you prevent DoS attacks and keep your site up when you are featured on CNN. In short: It helps you keep your site up at all times. And they do it for A LOT of websites.

On March 3rd, the entire CloudFlare network went down. Bad. This was bad for a lot of people (a lot more sites than the Redbus example), however CloudFlare managed to keep their customers by being as transparent as possible, and by posting a very detailed explanation of what happened. They even put together a video showing exactly how their entire network went down — all that the very same day of the outage, and they even included an explanation on how it will never happen again.

I call this a good reaction. They managed to keep most of their customers for two reasons:

  1. Their service is usually irreproachable (and they deal with some attacks that would burn your site to the ground).
  2. Customers felt confident that they did anything they could to fix the issue as fast as possible, and that it should not happen again.

So instead of flying away most of their customers stayed but took extra precautions because they got burned once. They might have signed up for a backup CDN or other measures so that CloudFlare would not be a single point of failure for them anymore.

It applies in your job. We are looking for honesty during interviews

But I did it right! The letters moved overnight, not my fault...

But I did it right! The letters must have moved overnight?

Screwing up is human, it happens. I am a systems administrator at Indeed, and sometimes I have to ask my coworkers for help saying “I broke that and I don’t know what I’m doing, can somebody help?“. The hardest time to ask that is at 3AM when a full datacenter goes down and you have to call a more senior coworker in a plea for help. (Thankfully, it doesn’t happen a lot, it’s really for the big bad stuff.)

When we interview people for a position on our team, one of the things we are searching for is honesty. If we can find a weakness in the candidate we interview, we’ll poke around it to see if s/he will try to invent random stuff to trick us or admit that s/he will need help on that. When you work in production, you don’t want to try random stuff you saw on Stack Overflow the night before and try to hide the fact that you took the site down for 2 hours the next day.

Take responsibility for what you do (good or bad), it will make you grow as a person — but don’t screw up too often either or make the same mistakes twice, heh.

09. May 2013 by Alban
Categories: Behavioral difficulties, Technical difficulties | Leave a comment

Let’s build a business with our laziness

Like most of the lucky people who have a daily internet access, I procrastinate. Way too much.

You hold more mathematical power than Pythagore could ever dream of

I can't say the battery life improved since my Nokia 3210

I can’t say the battery life improved since my Nokia 3210

Never before has humanity been able to share as much information and knowledge as this past decade, and the computing power in your pocket used to require a computer the size of a football field.

But most of what we could do with this amazing technology surrounding us is wasted by one thing: we are lazy. I’ll take an example I know pretty well: me. I started learning Python (see “How PHP ruined my life as a software developer“), and for this I am reading some books after work and working on some projects. I could be a lot faster though.

hmm-should-i-do-something-productive-today-comicEntertainment is valuable. Laziness is not.

When I am off work, I usually still have at least 6 hours before going to sleep. I often go for outdoors activities or social things with friends. This is definitely not helping me learn Python, and I end up not being at home a lot. This is not being lazy. This is just part of my life; I’m a very active person that needs to go out and do things all the time.

My laziness shows up when I’m at home. When I cook/eat, I usually allow myself to watch a TV-show. Relaxation, not laziness. The show lasts about 20 minutes, which might not be enough for both cooking and eating. Well, let’s watch another episode, that’s 40 minutes. Still acceptable.

Then, if it is not bed-time right after eating (which would mean that I have had a fulfilling evening), I should start working on Python things. Or if I’m too tired, read a book, clean my bike, do my laundry, anything.

I inevitably end up watching cat pictures

I am not implying that I have a brilliant mind.

I am not implying that I have a brilliant mind.

The last thing I do before turning the light off at night is often to browse 9gag. This site being addictive, I end up spending 30+min instead of the 5 minutes that I should allow myself to browse it.

If I could restrain myself and actually stay in this 5 minutes limit, I could get more sleep. Or if I don’t need more sleep, I could take this extra 30 minutes to read a book about building a business, or about Django, or about sysadmin stuff. Or a regular book, to make my imagination work.

No, I just look at this type of stuff. What the hell, brain?

The global cost of procrastination

Let’s say that I do this every day, for a 20 minutes average. Rounded down, this is about 2h a week. 104h of my life blown away every year — time I could have used to be productive.

Watching movies, going to a concert, chatting with friends in a bar… That’s entertainment. Watching fail compilations for 104h is more brain alienation. Nothing positive comes out of this: my brain would relax a lot more if I was sleeping, and I would be more entertained doing something else. No productivity on the horizon either.

I am also pretty sure that I’m a “small” procrastinator with my 20 minutes a day. Multiply this 104 by the number of lazy internet users, and the ratio that the average internet user spends on these types of sites: We could build a multi-million company with all these man-hours. Or entertainment centers all around the world.

Go to bed now instead of browsing Youtube. Or start a business.

26. April 2013 by Alban
Categories: Behavioral difficulties | Tags: , , , | Leave a comment

← Older posts