Reed Hastings, NetFlix CEO, gets it right

Netflix CEO Reed Hastings

Here's a video I shot at a NetFlix company party way back in the early days, back around March of 1999.  Kind of fun to relive the glory days of the Internet bubble.

At the end of the video, you can see a brief bit from Reed Hastings, CEO of NetFlix, when I asked him about the future of the company.  What's remarkable is that he knew this was the future way back then, and he knew he'd have to wait until the time was right to realize his vision.  Now we're here and he's done it.  Congratulations Reed!



That's the kind of quality you want in a CEO.

The Chief Technology Officer of a Software Startup

During early phases of a software startup, you don’t want to be management top‐heavy, yet it’s important to bring on senior people for continuity as you scale the organization up because these people can and should set the culture for the company going forward.  Once set, cultures are harder to change.  At the beginning, you need “player / coaches”; that is, people who can be an individual contributor, manager, and leader. The CTO in such a company should initially multirole as needed the roles of chief architect, lead programmer, and VP of Engineering, eventually replacing these with full time positions at the right time.


It’s important for the CTO to select the right technology platforms and tools early on in a startup, as well as play a key role in hiring the team and setting up the team culture, development infrastructure and development processes that can scale.  To be successful in doing that, the CTO absolutely must be a visionary technologist and quality coder with the ability to learn new technologies as needed.

A common view of the CTO role is that they do not need to have strong technical skills.  John Wren Medcof shows this well in this picture on the right from his web site.  I disagree with this model, and think that a CTO needs to excel in all these domains.


While the CTO is responsible for the technology, the continuous product and project management decisions of how to balance milestones, resources, and features as development progresses must be a cross functional team decision. No one person has all the knowledge to do this. In larger organizations, you form a cross‐functional product team to balance the three factors (time / features / resources). However, in early startup stages, the executive team may need to actively multirole this until sufficient staff is in place to offload this burden. I use the word “burden” because it takes some investment of time in order to make the best call to balance these factors. 

If the executive team is also playing the role of the cross‐functional product team, the executive team meeting should be kept completely separate from the cross‐functional team meeting. The two meetings often have different people leading them as well. Usually a CTO or product manager leads the product team, but that doesn’t have to be the case.



The CTO should know when it’s time to hire people, and what kind of skill sets are needed. When the size of the engineering team hits a certain point – perhaps around 5 people and still growing, another layer of management may be introduced – this typically reports into the CTO but this is not required. The next manager brought in should either be at the Director or VP level, and that person may need to be a player / coach until the team grows to a size where the manager can’t simultaneously write code and manage.  As the team grows larger, additional layers of management should be brought in.  There are "phase changes" of sorts that an engineering organization (and the company as a whole) goes through when the total size hits certain ranges.  A CTO should be prepared to  help the company through these transitions.

So to summarize, the CTO may well start off as a strong individual contributor, but that function should gradually be reduced as the team grows in size. With that said, in order to maintain his or her edge and to retain “street credibility”, I believe a CTO must continue to develop and write code, as well as facilitate the architecture, and mentor others on the engineering team.

Most of the CxO’s on the executive team must play both an internal‐facing leadership role, and an external‐facing role facilitating investment and sales. The CTO is no exception. Initially as the company is getting going, probably more emphasis should be placed on the internal role. As the company grows, the CTO should transition to a having a larger external impact, through multiple channels. The CTO is often brought along as a “big gun” for important sales, in order to talk tech with the customer’s tech people. The CTO is usually part of the investor pitch team as well, as investors need to know that the company has what it takes to deliver the goods. In addition, the CTO should develop an external reputation for technical excellence and thought leadership, in the industry and his CTO peers in other industries. This is accomplished through several mechanisms such as blogging, articles and press releases, and talking at industry events.

I’ll close by saying that a major challenge of being a top CTO, is maintaining that technical edge that allows you to be a “player” in the player / coach model, yet spend enough time on the external facing role and continue being a thought leader. You can’t do this by just coding a few hours every couple of weeks. You need to sink your teeth into some problems.

Passion, Joy, and Software Startups

Passion and joy are two of the most important parts of life, so as a software developer I think it's important to understand how to create a software career that brings us the most passion and joy.  As this eHow article on How to Follow Your Passion says:

Remember that following your passion is the most important thing you can do for yourself when it comes to your career. The majority of people are programed to not follow their passion when it comes to their career because they still fear the unknown. But following your passion early on is a must if you want to live a succesful fulfilling lifestyle.

One of the best things about America is that our culture supports entrepreneurs who build their careers by following their passion.  Although there is an interesting counter-point to this: The average Peruvian is 3.5 times more likely to start their own small business than the average American.  Maybe I should start my next startup in Peru.

According to the respected publication The Economist, between 1996 and 2004 America created 550,000 small businesses every month.  Wow!  And one of the best things about being a software guy in the Bay Area of California where I live, is that we are home to so many software startups.  In the SOMA area of San Francisco where I've worked at several startups, the number of startups has just started surging in 2010, according to The Wall Street Journal.  As SFGate puts it:

In San Francisco's South of Market neighborhood, Internet-based software startups are thriving despite the worst downturn to hit California since the Great Depression.

Some of you at this point will be saying, "but how many of those startups survive?"  The table at right shows the percentage of startups still in business up to 10 years later.  This data is from the Bureau of the Census produced for the Office of Advocacy of the U.S. Small Business Administration.  Another data point comes from Scott Shane's excellent book, The Illusions of Entrepreneurship.  In it, he states that one third of startups are successful after seven years, which agrees pretty strongly with the U.S. census data.  Scott characterizes this as "only one third", but I think of it like, "Holy Shit!  That's good odds.  Just have to try a few times and I'll get it.  Might even get lucky and hit my first one out of the park."

I believe following your passion is part of American culture and especially Bay Area culture, and is part of self-actualization.  This is pretty deep spiritual and philosophical territory.  Earlier this year I blogged over on my personal blog about the relationship between self-actualization and destiny.  Interestingly, Scott Shane states that the main reason people start a small business is to avoid working for others.  I think that perhaps it's harder to self-actualize if you're following someone else's business.

When you reap the reward of joy from pursuing your passion, you may also accumulate some amount of wealth - perhaps even a lot of wealth.  It's easy to forget that the value of following your passion is the joy you reap from doing just that - not so much from the wealth you accumulate.  In fact, one famous study suggests that lottery victims experienced less overall happiness than accident victims.  I take that to mean that wealth in and of itself doesn't bring joy or happiness one bit.  It comes from following your passion.  I love this quote from Health Guidance in the article True Wealth Will Make You Happy But It Must Be "True":

True wealth lies in defining ourselves by who and what we are, not by what we do or do not have.

Many software startups that started with lots of passion become successful, so they grow big and then they have a problem as they grow larger: Innovation and the entrepreneurial spirit flicker out.  How do they keep the innovation and entrepreneurial spirit alive as they grow?  Google has made famous (but did not invent) the idea of reserving a certain percentage of everyone's time for innovation - their famous 20 percent time.  Google says this in their section on an engineer's life at Google:

We offer our engineers “20-percent time” so that they’re free to work on what they’re really passionate about. Google Suggest, AdSense for Content, and Orkut are among the many products of this perk.

I think the key part here is giving people a way to "work on what they're really passionate about".  By implication, the other stuff they're doing, well - they're not so passionate about that.  Hmm... That kinda sucks then, doesn't it?  That means 80% of your time is spent doing relatively boring stuff, and that to me seems like a high tax.  I'd kinda like it if I spent much more of my time working on stuff I'm passionate about.  Where can I do that?  And the fine print of Google's employment contract says they own 100% of any vision I create in my 20% time, so I'd like to keep at least some of that for myself please.  Where can I do that?

I believe there are at least two routes to finding more passion.  One route is to find someone else with passion that's out there building something, drink from their cup of passion (which should be overflowing), and then you will join with others in this way.  If you're following this route, I think it's important to find a group of people with passion for what they're doing - don't settle for people that don't care so much about what they're building.  If that kool-aid cup contains apathy, you will most likely have a bitter taste in your mouth.

Another route to passion is to realize your own vision in the form of a startup business, based on an idea you're truly passionate about.  How do you know if you're passionate about something?  I like the way Ryan Hoback puts it in his posting The Passion Behind Business:

When I get excited about a new venture that I have been developing, I feel the excitement run deep through my body. A feeling of invigoration comes across me. When a new idea comes to my mind, I begin to get happy like a little child opening a present on their birthday. You see, this is passion, and this is what successful business organizations are built on. Passion. Passion is the driving force and inner desire among entrepreneurs and leaders across the world which makes you want to continually pursue going above and beyond.

 Cash Miller puts it this way in his article Why is Passion in Small Business Necessary?

The number one asset a small business owner and entrepreneur needs to have is passion. Passion for your business is an essential ingredient when building a business. It can help to energize you when times are tough and it can help fire up your employees. Passion is what makes people believe. Do you have that passion?

So maybe you do have all the passion in the world, but most likely you don't have the personal resources required to start your own business all by yourself.  It takes a lot of time, which means you can't use that time to earn income, which means you must have a means of sustaining yourself while you're building your business.You can't eat passion for breakfast.

In addition to your own time, you must invest money in one way or another, either by eating into savings, or in the form of opportunity cost - lost opportunities for earning income, or in the form of loans, or an angel investor.

In addition to investing time and cash, if you're doing something really big, you will need to team up with other like-minded entrepreneurs as partners.  You will all need to drink the same passion, and it will take time for you to all get on the same page and share a passion for the same vision.

When not to use Agile

When building products with teams of people over the course of months, I think the Agile project management approach works best.  I have my own customized Agile approach I like to use that I'll likely describe in a future post.  But first, this posting describes when not to use Agile.

I've been noticing an increase in the number of extremely short, tactical coding exercises to deliver urgently needed functionality within hours or days.  More and more, I see people doing things like: throwing together a web site, building a prototype, creating a custom ETL solution, creating a new kind of widget, fending off a hacker attack, or creating a new report.  Sometimes the code may only be used once, or it might be used for years.  Often times these coding exercises must be delivered as quickly as possible, and with no advance notice.

I've found the best way to deal with these situations is to immediately create a team composed of the people that need to work together to deliver this.  Have them meet and figure out who's going to do what and when, and then they go off and do it.  No sprint meetings, no burn down charts, no stand up meetings - the team just gets the job done.

The challenge here is that the set of people that are working on these types of urgent tasks may largely overlap with the set of  people working as part of an Agile team, and they will most likely be in the middle of developing work that they've committed to delivering by the end of the sprint.

I think there are a few solutions to this problem, each with pros and cons.

First, you can reserve a certain total percentage of your Agile team's velocity for urgent tasks - say 20%.  This basically comes off the top of the total team velocity that you start with at the beginning of a sprint planning session.  At the same time, you identify a few stretch tasks in the sprint - bonus features that are either in the next sprint or that would be really nice to have.  If you don't use up your 20% reserve, you can tackle some of these bonus features.  The down side of this approach is the extra cost for context-switching mid-sprint; it takes extra time to switch between a person's sprint tasks and their 20% urgent task.

Second, you can just make sure that these two sets of people don't overlap - reserve different people for the short, tactical projects and manage that pool of people differently (but also probably not Agile).  You can periodically swap people between these two sets for cross training purposes.  The down side is that this probably requires more total people, so it's much more expensive.

Sustainable Software Engineering

Something bad happens to many software development teams somewhere around the 4th or 5th year of life of their code base: Quality and productivity take a nose dive, following an ever increasing trend of problems.  I call this the "Developer Duldrums" (with apologies to Norton Juster) because it's a sad place to work.  It's insidious because the team is usually building software the same way it always has, yet they've lost their mojo.  Why?  The answer is that the code base can no longer sustain development.  Enter sustainable software engineering practices.

Sustainable software engineering consists of:

1. Automated functional testing.  I like to bias most of my QA team to writing automated functional tests.
2. True unit tests (not functional tests masquerading as unit tests)
3. Keeping your code bug free right from the beginning
4. Relentless refactoring
5. A solid and always-evolving architecture that is well internalized by the team
6. Properly componentizing (with tests on the interfaces) and isolation of subsystems in your architecture so that you can re-write sub-components when needed without touching other parts of your code base
7. Little overtime

Sustainable software engineering is usually not practiced because there is a cost associated with it, a cost which has a return that may be a year or more down the road.  However, if you haven't paid your dues and you get yourself a few years down the road, the chickens will come home to roost.  At that point, you're looking at rewriting your software - that's something you want to avoid because of the large cost and risks involved.


Counterpoint: In my experience, I've observed that technology trends move fast enough that approximately every five years, there may well be sufficient benefit to re-architect your code on top of newer technologies to justify a return on the investment required to do so.


So it may happen by luck that the Developer Duldrums curve may drop off close to the time you'd want to rewrite your product anyway because of advances in technology, so sometimes that's your way out of the Duldrums.  You kill two birds with one stone.  However, I wouldn't want to rely on such a coincidence happening.


Shorter projects benefit less from sustainable engineering, simply because you don't make your return on the investment because you don't have software you need to sustain.  So the amount of sustainable engineering you practice should be proportional to the project length or expected life time of the code base.


Sustainable software engineering isn't always possible to practice.  I would guess that there is approximately a 25% cost overhead for doing so.  This is due to the time it takes to write quality unit tests, refactor your code, and fix the important bugs as they occur.  (Counterpoint: The use of automated unit test generator tools such as Agitator from the beginning of the product development life cycle may reduce this tax.)  Often times, your primary objective is to reach your customers as quickly as possible to validate your market and product.  To do so, you could get to market faster by skimping on the sustainable engineering stuff.  That's certainly a valid argument.  But if you go down that route, you should be prepared for the chickens to come home to roost at some point.

Since I wrote this blog posting, I discovered the book Sustainable Software Development, by Kevin Tate.  I asked Kevin what he thought of my blog posting.  In his reply, which you can read below, he mentions something really interesting about proper componentization (which resulted in my point #6 above - thanks Kevin!):

I like the post and agree with the main points.  One thought is that you mention a 25% overhead, but in my experience people need to recognize that you typically get that 25% back (and more) through the knock-on effects of, for example, having reduced QA / manual testing and decreased effort behind release cycles.  

Another thought I had is that you touched on the need to periodically rewrite based on the latest technology.  That's where you might want to add one other "critical element" to your list: the need to componentize (with tests on the interfaces).  I talk about it indirectly in my book, but since I've written the book it's become increasingly obvious to me that this is another element of "secret sauce" because it allows teams to selectively rewrite their product as they go without having to throw out the entire system.  The rewrite from scratch scenario doesn't succeed very often, if indeed you even get the business support to do it!

Building high quality software fast and cheap

James Currier at Ooga Labs posted a blog entry that starts as follows:

"I’ve heard people tell me “We can build product fast, good, or cheap.  You can’t have all three.  Pick two.”  I believe this is a corrosive mindset, used by bureaucrats to justify mediocrity, or used by people who are afraid of failure to set the bar low enough so they feel comfortable in their daily lives."

I disagree with James and think you can only lock down two on a large, complex software development project.  While it's possible to build good software fast and cheap, I think that happens only under certain circumstances where the engineering risk is inherently low.  Also, sometimes you get lucky and nail it - I've seen that happens a few times. Projects having any of these characteristics generally can be delivered fast, good, and cheap:

1.  Young code base (generally a few months of development or less)

2.  Small team size (generally 1 or 2 senior developers, sometimes up to 3 or 4 if things are going well)

3.  Relatively simple feature set

As you get to bigger teams with more complex software that takes longer to create, your risk goes up.

The underlying principle at work here is "shit happens" - sometimes an engineer quits or gets sick, or you discover a fatal flaw in a library you're relying on and you need to retool, or you discover that a feature doesn't work as designed and you need to redesign it, or the market changes and you have to react.  When such a thing happens, the bottom line is that it's usually going to take you more time to do what you wanted to do.

So let's say such a thing happens AND you hold all three values fixed.  You can deal with such unexpected events by building in a "slop factor" to your schedule, so you eat into your slop time.  But essentially what a slop factor does is pull in the release date by saying "we expect this much shit could happen".  So you really are slipping in a way so the "fast" part is actually varying.  Or you could ask your team for overtime - that's one option.  But that's really tweaking the "cheap" part because you are putting more time in - if you paid for that hourly, your cheap just went up.  You could shave your feature set and simplify things and still ship, but then you're tweaking the "good" part.

Sometimes bigger shit happens than you budgeted for with your slop factor- we all get constipated from time to time - and if that happens, then what?  It's naive to believe that big shit won't ever happen.  Then what, if you hold all three fixed?

I call these three values by different names that more closely connote the values being managed:

1.  Release date
2.  Features & Quality
3.  Resources (people and hardware/software)

My basic form of project risk management is to look at these three values as dials that you can twiddle or tweak at the beginning of each sprint.  So I've folded this form of risk management into the way I practice Agile project management, by constantly reassessing the feature set, team velocity and composition, how the product is shaping up, and the importance of the release date relative to other business priorities, which may be more fluid than we'd like.

Why perform technical due diligence prior to an acquisition?

Michael Sisco put it well I think: "Conducting thorough [technical] due diligence can make the difference between a successful merger and a company's collapse."


If you're a high tech company looking to acquire another high tech company, please perform technical due diligence as part of negotiating the deal, ideally prior to signing a term sheet.  Here's what could happen if you don't.

First, you won't understand the costs, timeline, and resources required for delivering the promised integration of the newly acquired company.


Second, you won't have a good understanding of who to keep and who (if anyone) to let go as part of a Reduction In Force (RIF).  You won't be able to develop a new organizational structure because you haven't worked out what you're going to do and in what order.


Third, there is some value in the technology of the acquired company.  The value is significantly correlated with the quality of the code and architecture, and whether you have to rewrite the product code or not.  This should figure into the overall acquisition cost negotiation.


Fourth, you won't be able to optimize your own product development schedules to account for the possibility of the coming acquisition.  It's often possible to re-order development tasks to intelligently account for the possibility of the coming acquisition, without incurring cost if the acquisition falls apart.  Or you might pause or slow-roll hiring certain roles until you know how things are going to shake out.


There is always time to perform technical due diligence.  More information is always helpful.  For smaller companies, you can usually do the whole thing in a week.  Get the book Technology Due Diligence if you don't know how and want to do it yourself.  If you don't have the time or skills to do it yourself, you can hire a firm like blackduck to do the technical due diligence for you.


Since I manage the Chief Technology Officer Exchange over on LinkedIn, I asked some CTOs what they thought of the practice of investing in or acquiring other companies without performing technical due diligence.  Here is one response I find particularly interesting from a CEO in the group:

This kind of acquisition is a lot more common than it used to be. I'm seeing a growing number of companies decide they don't have to worry too much about moving software forward from its current state. That leads them to conclude they can do all sorts of things that seem like bad practices. Acquiring without diligence is one. Not taking care of the developer team is another. 

I haven't seen much evidence that these formulas are successful, it's just that I'm seeing companies try them more often. 

Another surprising one is for VC's to invest without proper technical diligence. This happens pretty often and has been going on a long time. They get stung by it periodically.

In general, most non-developer types have no real idea how to manage, value, or evaluate IP assets.

Multiroling

I'd like to introduce a new term into our team culture / process lexicon: multiroling.

Multiroling.  Noun.  The carrying out of two or more roles at a time.


When assembling a team, you first have to think about the roles you want on the team, and then you go about assigning people to those roles.  You ignore formal HR role names when doing this - you pick the best people for the job.  When a person is assigned to a particular role, they act in the capacity of that role rather than in their formal Human Resources Management Title.  So when I, for example, act in the capacity of product manager, then I do not exercise any authority that my official HR title would give me.

Multiroling is one of the key techniques I use when building a team.  The problem is that you usually don't have an instant, fully functional team.  You build it up over time, usually a few months.  During that time, you don't have all the skill sets needed, but you still want the team to build stuff.  So you create place holder positions for people you haven't hired yet, and you multirole other people into those empty positions until you fill them.  As the team grows, members may shed roles, to be assumed by new team members.  This creates the proper culture and process for the team so that you can grow quickly.

In larger organizations, if a person assumes multiple roles, they may need to engage in matrix management (also known as "dotted line" management) while assuming roles that cross managerial boundaries.  In addition, you may need to become fluent with multiple cultures.

During the course of a typical day, I assume different roles from hour to hour, depending on the type of meeting I'm in or what team I'm interacting with.  Here is the actual current set of roles I'm multiroling:

1. Interim-vice president of Engineering / Executive team member
2. Director of Engineering
3. Product Manager
4. Architect
5. Project Manager
6. Senior Salesman

In order to pull this off successfully, I've found that it works best to be clear about what role I'm assuming before starting a meeting or discussion or email.  I might even start a conversation with something like, "In the role of Architect, I'd say that..."

This is one of the dynamics that makes start-up companies so explosively successful - they are able to multirole.  What happens as an organization increases in size, though, is that people tend to be pigeonholed into one role only.  Some people really like multiroling, so if a company loses its multiruling culture as it grows, then those people move on.

The primary challenges with multiroling are two: First, you simply may not have enough time to do the new role justice - something has to give somewhere.  Second, you may not be well trained in your new role (although maybe you're the best your team has got).  You need to assess whether your skill level is high enough to do the job justice.  It doesn't make sense to take on a multirole you can't really fulfill in any meaningful way.

Perhaps this points to a short coming of our traditional HR organizational structure.  Perhaps we ought to adopt a more flexible HR title that reflects the multiple roles that people assume over time.  Oye, we'd have to print so many business cards as we added and dropped roles over time!  So that's why my current business card actually has the role name "Omelette chef" on it.  Really it does!

Team building

I've found that most large business goals are unique, and require assembling the appropriate resources to achieve. These resources start with people, and also include the equipment and other assets needed to achieve the objective.

With respect to organizing people, you look at the people you've already got on board in terms of existing employees and consultants, and you also look at new employees or consultants or outsourcing in various ways. The choice of which type of human resource to use often depends on the amount of time they'll be needed, how specialized, rare, or expensive their skills are, and whether the knowledge built during the process of doing the work is something that would add value to an existing strategic knowledge base or cultural asset.

In many cases, you must recruit existing employees or consultants that are already busy doing something, so getting them to take on these new tasks requires an understanding of how work is requested, specified, assigned, performed, tracked, and valued within an organization. This requires an understanding of both the processes (e.g. feature requests, hiring processes, purchase orders, help desk requests, budgets, etc.) and culture of how to get things done. It also requires an understanding of what motivates each and every one of the individuals involved, because those motivations need to be aligned as much as possible with an emerging commonly felt group motivation for getting whatever it is achieved.

Every team has a life span. Sometimes that lifespan is very short - a few hours - and some times it's years. Or anything in between. Sometimes the teams grow and shrink over time as new needs evolve.  It's important to know when to grow and when to shrink a team.

It would be a mistake to view a team as the sum of its members.  In the best teams, something magical happens when the team is working together well - a team spirit emerges that is palpable - you feel it on the team.  It's what makes working on one team feel really different from working on another team.  The members feel everything is clicking, they're in the groove.  And the whole company knows it.  They are able to deliver best in class, innovative software that solves some of the toughest engineering challenges around.

On such teams, they are self-organizing.  Team members instinctively know when and how to work with other team members efficiently.  Leaders naturally emerge when needed to solve problems with the support of others.  Overtime is rarely needed.  Deadlines are usually met, or if not, for good business reasons that have appropriate management / executive support.

Software engineering is different from other kinds of engineering

When a company's primary product is not software, but rather contains software as a sub-component of something larger such as a piece of hardware, I have found that the management, processes, and culture of such companies generally do not support innovative, high quality software development.  Some online internet services are similar, even though they are entirely enabled by software - since they aren't selling software, but rather a service, the support for innovation can be weak weak.

What happens at first, in such cultures, is that the software developers are left alone to figure out how to make things work, and they pull it off.  So management assumes that its approach in terms of management, processes, and culture must be working.  However, what inevitably happens is that things gradually, eventually start to fall apart in terms of productivity and quality.  At this point management often thrashes about trying various solutions to the problem that may be partly effective or not, not realizing that *it* in fact is the problem and that management is what needs to change.  Management's hubris in such a company can often get in their way of seeing themselves as the root cause.  This can create a morale problem for the software developers working in such an environment.

Management might try to bring in "process" in some form - importing processes from other organizations in the form of specification processes, project management tools, version control and configuration management systems, testing tools and procedures, and so on.  This can be helpful in some ways but isn't a silver bullet.

Management might try to bring in "culture" in some form - "Everyone show up at 8:00 am from now on and no more telecommuting even when you're working overtime," for example.  Yeah, that's going to go over well with software developers.  Not.

Management might try to bring in new management - a manager that supports upper management's values.  However, if management fundamentally does not understand or value software development, then it will bring in a manager with that philosophy and the result is that the manager will be a proxy for the culture and processes that management wants to inflict on its software engineers - "We just need someone to make them follow the process."  Yeah, developers will like that.  Not.

Managing a high quality team of productive software engineers is so fundamentally different from managing other kinds of engineers, it can lead to upper management's mistaken assumption that they can apply other engineering management practices and experience that, in the end, is not effective.

Unit Testing

I've noticed that the phrase "unit testing" means two pretty different things.  It can mean:

1. A developer tests their own code as best they can by itself.  Others might call this "white box testing" or even "white box system testing", but my current client calls this unit testing.  Since the testing is generally not automated, there are no automated test results.  Test results must be typed in by hand, if indeed test results are even reported with this method (they are usually not).
2. A developer writes automated unit test code, typically using a unit test framework and using mocks as needed, to test their interface and achieve maximum code coverage, with objective pass/fail results.

I'm currently working with a client that uses the definition in #1, and I've had to broaden my definition of unit testing to incorporate it.  There is so sense asking my client to use what I think is the correct definition.

I'm a great fan of the second kind of unit testing.  It gives me the confidence I need to refactor relentlessly, constantly throwing out old code and improving what's there.  That's one of the secrets to keeping productivity up over the long run.

But sometimes there is no long run.  Sometimes the projects are six week tactical exercises, after which the code is done.  In those cases, unit tests would be a waste of time.

Sometimes it was simply not possible to invest the extra time to write unit tests, because there is an initial investment in time with a payback period, and sometimes you can't afford the investment of time.

But some times those six week tactical exercises get cloned, or turned into a six month exercise.  But you never wrote unit tests from the beginning.  And the thing about unit tests is, that if your code base doesn't already have good unit test coverage, it can be quite time consuming to add after the fact.  So time consuming, that you won't do it, and you'll end up with a project that you wish had good unit test coverage, but it doesn't.

Often times, unit testing is introduced during a "re-architecting" phase, which generally seems to happen about once every five years.  That's a topic I may blog about some time - the re-architecting phase of a product life cycle.

Making a software shit knife

Attachment

When I use the word attachment here, the closest meaning comes from the Sanskrit word Upādāna, which is generally translated as attachment.  I have my own related view of attachment.  Although it is alleged to be one of the primary causes of human suffering in some Eastern traditions, in this context the only suffering that could come from attachment to software is making the wrong choice about a technology decision.  I guess you could suffer from that.

Making a shit knife

Geeks like staying abreast of the latest cool shit.  Usually a geek will start their career by hitching their wagon to some piece of cool technology shit that they develop some real depth with.  New pieces of hot shit come by, and the geek is happy to add those to the rest of his cool shit.

When a new problem comes up, the geek looks in his bag of cool shit and pulls out a piece and says "I've got just the cool shit for this."  And he applies his piece of cool shit to the problem and all is good.  In fact, sometimes a geek can get so good at fashioning anything out of his cool shit, he can create something like the infamous Inuit shit knife.  Now that was some serious shit.

The problem of attachment to software

Lucky that Inuit was in some cold climate, because otherwise he probably wouldn't have managed his escape.  And like him, sometimes I am lucky.  But sometimes I just don't have the right cool shit in my bag, so I'll find something that looks close, but it's not the right shit because the right shit isn't in my bag.  When I'm attached to a particular language, I'll choose that language for projects over other languages, even though another language might be a better business choice.  Sometimes I get attached to new technologies I haven't tried yet, because they just seem so damn cool, and then once I learn them I just want to use them everywhere.  Sometimes I've learned a particular pattern for how to solve a problem, and I'll just apply that pattern even though it's not the best choice.  Sometimes a bug will elude me because I'm attached to how I think the code is supposed to work.  In other words, I can let my own attachments interfere with making the best business decision.

Software attachment goes beyond what we have as software developers though.  We're attached to the latest version of our favorite programs, that cool screen saver, the cool web thingy at work, and on and on.  If we have to use something different, we're moving into new territory, the unknown - and that can be scary or stressful for some people.  It is an opportunity for others.

I'm still trying to figure out how to best address this problem in myself, by trying to identify when I am overly-attached to a particular technology to the point where it's getting in the way of making the best business decision.

Gut instincts

This is a story about how I knowingly voted with a group to make the wrong decision.

As a group, we were evaluating several alternative technology stacks for building a healthcare application.  We listed out all the qualities that were important to us in the stack - things like quality of development environment, availability of third party tools, and so on.  We gave each of these qualities a numerical ranking of importance.  Then we evaluated each technology stack for each quality we'd listed, and ranked how well that stack supported that particular quality.  We had a handy spreadsheet that calculated the total for each technology stack.  As a group, we all agreed to all the numbers on the spreadsheet.  The spreadsheet told us which stack won - and that's the one we went with.  Bam, done.  We could even explain to others why we'd chosen this stack.

Yet the spreadsheet gave us the wrong answer, and the group didn't figure that out until several months later.  My gut feeling at the time also told me we were picking the wrong technology stack.  I tried to explain why the choice was wrong, but because I couldn't express my gut feeling in this spreadsheet, my explanations were not effective.  I even tried manipulating the weights of each quality, and even added new qualities to the list to try to get my choice to come out as #1, but I couldn't do it in a credible way.  I think a few other team members were in the same situation too.  But we ignored our gut feelings and went with the #1 on the spreadsheet.

Several months later the group realized they'd picked the wrong horse and started over again with another one.  How is it we chose the wrong technology stack?  We consciously tried to pool our knowledge in this spreadsheet.

Maybe the problem was that this spreadsheet approach simply excludes gut feeling, right-brain, non-rational thinking from the process.  So we even added a row to the spreadsheet for "gut feeling".  Really, we did that.  That didn't help the spreadsheet come up with the right answer though.

Sometimes it's a mistake to over-formalize the decision making process of a group.

Should a group leader/manager/architect have veto power over whatever decision a group comes up with?  Would that have helped this group by making the right decision to begin with?

Should you always trust your gut instinct?  Probably not.  But you should seriously pay attention to it.

Knowing when to fold

When I was at Oracle during the 1990's, I got to work on this huge framework called Sedona. After Oracle invested several years and hundreds of people working on it, I joined the project and was building the entire web front end for the tool (the web was brand new back then, remember).  Then,  I read one day in the press that the project I was working on had been canceled.  I left Oracle that day, because I felt like I should have heard that my project was canceled from inside Oracle before I read about it in the press.  I didn't disagree with the decision, but I did feel like the management culture at Oracle was not healthy for me spiritually at the time.

The architect for Sedona was a very bright guy named Chris, who went on to Microsoft to help create the COM+ distributed transaction framework, and then the wildly successful Common Language Runtime that's part of Microsoft's .NET framework.  I had lunch with him a few years ago when I was working there as a Principal Development Manager, and I'm really glad he found an outlet for his professional creativity.

Chris and I have never talked about this so I don't know what he'd say, but I think some of his Microsoft work bares the marks of "doing Sedona right the second time at Microsoft", and I really see his Oracle and Microsoft work as part of a continuum.

Oddly enough, in my current consulting work, I've had an opportunity to learn this enterprise computer system called Pivotal.  It's pretty close to what Oracle was trying to build with Sedona, and I wonder now whether there were any former Sedona people on the Pivotal team.

UPDATE 4/14/10: I talked with Martin Stiby (a really bright and very nice geek) at Pivotal and he told me that the design did not come from Oracle or Sedona, just to set the record straight.

I think one of the hallmarks of a great CEO is that they have a good track record of knowing when to fold and when to double down.  Sailing metaphors were really popular for a while in management consulting.  When is the wind going to shift a certain way?  How does the team on the boat adapt as a group to a constantly changing wind and then win a race?  Maybe they're not always right, but they're right enough to win.  Given Larry's track record, I would give him the benefit of the doubt on this one.  How did he make this decision?

I think knowing when to persevere, and when to call it quits is a key skill that is transferable to many parts of my life.  In software development, I see this in development, debugging, testing, management, user interface design, and more.  Often, such decisions are made by individuals and no one may even know that a decision was made. But also these decisions must be made by groups, and it's here where culture can play a critical role in helping the group make the best decision.  Whether it's the board or the executive team or your daily engineering stand up meeting, culture helps groups pool their collective knowledge and make wise decisions about when to fold.

My favorite bug

This is the story of my favorite bug. I created this bug on an early version of the NetFlix site, over a decade ago when they were a relatively small, unknown start up. Now I stream NetFlix on my XBox, and DVDs in the mail seems... so... last century.

Way back when NetFlix was a tiny start up in Scotts Valley, CA, I walked down to their offices from my house up the street, and walked out with a job as a senior web developer. They soon moved just over the hill to Los Gatos, CA, where I got to implement this fabulous bug I want to tell you about.

When you browse the NetFlix web site and look at a movie, you see all this information about the movie such as the description, actors, director, and so on. NetFlix used to get this data from the Internet Movie Database. I don't know if they still do.

Anyway, my task was to set up a process for automatically getting the latest content from IMDB and bringing that into the Oracle database so it could be displayed to the user on the site. My basic approach was a standard "Extract, Transform, and Load" or ETL process. I set up a job to retrieve the latest data from IMDB via FTP, then I wrote a program to do some validation and pre-processing on that data, and loaded it into a few new tables I'd created in the Oracle database. Finally, I kicked off a big PL/SQL script I wrote to process the newly inserted rows by updating the actual movie reviews, actors, and so on - where they really lived in the database.

When I ran a test of the PL/SQL script against a test copy of the production database, I noticed that it took several hours to run, during which time the database was so over-taxed that actual end user response times would have been unacceptable. So I came up with an idea to process one row and then sleep for a few seconds, then do another row and sleep, and so on. That way the content would still be imported, it's just that it would take a few days for the process to finish. And more importantly, database performance would remain acceptable during the process. Sounded good.

So this passed QA and was put into production. Over the next day, people gradually began seeing the new site content, one new title every few seconds, and they were pleased to see this fresh content. What they didn't realize until some time later, was that each time my PL/SQL script updated a movie, it set the available inventory level on that movie to 0. This effectively took it out of stock as far as the web site was concerned, so that movie was no longer able to be rented through the web site. Over time, the entire inventory was being taken off line, unavailable to be rented. That was their sole revenue stream, mind you.

At some point before the entire inventory was destroyed, we figured out what was going on and ultimately ended up restoring an Oracle database backup and deleting the PL/SQL script I'd written.

Over the next few days myself and others worked to understand the root cause of what happened. How could this have passed through QA? Well it turns out that NetFlix used Oracle Financials, and that was running on the same Oracle database server. Oracle Financials was not present in the QA test setup. Oracle Financials saw this movie content update as essentially meaning a new movie was in the inventory, so its available inventory starts off at zero until you tell it how many you've got. So Oracle Financials was taking the titles out of inventory.

I had no idea Oracle Financials was even in the picture, and I guess our QA team didn't either. The bug fix for this was really simple once we knew how to get Oracle Financials not to view this as a new title. And eventually the new content got out on the site and all was good.

Over the next few weeks we talked about how we could prevent something like that from happening in the future. I'll never forget this really bright programmer there named Kho telling me that really good programmers just don't write bugs to begin with. Then he proceeded to show me all this bug free software he'd written. Once every few years, I seem to somehow write a huge block of code, and it just compiles and runs, bug free. And I am amazed. It can happen. I don't know if it's just luck or whether this can be cultivated. Maybe Kho is right.

What this blog is about

This blog contains my insights about the art of building software. This is an all-inclusive look at everything it takes to build truly great software. Sure I'll go over languages, technologies, platforms, APIs, movements, battles and wars, winners and losers, and so on. But just as importantly, I'll be discussing how groups of people can organize themselves and create a software development culture that consistently produces award winning, best-in-class, highly profitable and successful software. This builds a sustainable business model because you create a barrier to entry and competitive advantage in the form of your own unique software development culture. This is very hard to steal, and it's also amazingly hard to emulate. And it shines right through your brand equity and directly touches your customers, and your bottom line.

I'll also be touching on important industry trends and governmental issues from a global perspective. Europe, Asia, Africa, Israel, Russia and Australia (just to name a few) are all important places to monitor. Search engine wars, and the massive cat and mouse games played by hackers, script kiddies, and software developers world wide.