The art of building software

Sunday, December 19, 2010

Reed Hastings, NetFlix CEO, gets it right

Netflix CEO Reed Hastings
Here's a video I shot at a NetFlix company party way back in the early days, back around March of 1999.  Kind of fun to relive the glory days of the Internet bubble.

At the end of the video, you can see a brief bit from Reed Hastings, CEO of NetFlix, when I asked him about the future of the company.  What's remarkable is that he knew this was the future way back then, and he knew he'd have to wait until the time was right to realize his vision.  Now we're here and he's done it.  Congratulations Reed!



That's the kind of quality you want in a CEO.

Monday, November 29, 2010

The Chief Technology Officer of a Software Startup

During early phases of a software startup, you don’t want to be management top‐heavy, yet it’s important to bring on senior people for continuity as you scale the organization up because these people can and should set the culture for the company going forward.  Once set, cultures are harder to change.  At the beginning, you need “player / coaches”; that is, people who can be an individual contributor, manager, and leader. The CTO in such a company should initially multirole as needed the roles of chief architect, lead programmer, and VP of Engineering, eventually replacing these with full time positions at the right time.


It’s important for the CTO to select the right technology platforms and tools early on in a startup, as well as play a key role in hiring the team and setting up the team culture, development infrastructure and development processes that can scale.  To be successful in doing that, the CTO absolutely must be a visionary technologist and quality coder with the ability to learn new technologies as needed.


A common view of the CTO role is that they do not need to have strong technical skills.  John Wren Medcof shows this well in this picture on the right from his web site.  I disagree with this model, and think that a CTO needs to excel in all these domains.


While the CTO is responsible for the technology, the continuous product and project management decisions of how to balance milestones, resources, and features as development progresses must be a cross functional team decision. No one person has all the knowledge to do this. In larger organizations, you form a cross‐functional product team to balance the three factors (time / features / resources). However, in early startup stages, the executive team may need to actively multirole this until sufficient staff is in place to offload this burden. I use the word “burden” because it takes some investment of time in order to make the best call to balance these factors. 

If the executive team is also playing the role of the cross‐functional product team, the executive team meeting should be kept completely separate from the cross‐functional team meeting. The two meetings often have different people leading them as well. Usually a CTO or product manager leads the product team, but that doesn’t have to be the case.



The CTO should know when it’s time to hire people, and what kind of skill sets are needed. When the size of the engineering team hits a certain point – perhaps around 5 people and still growing, another layer of management may be introduced – this typically reports into the CTO but this is not required. The next manager brought in should either be at the Director or VP level, and that person may need to be a player / coach until the team grows to a size where the manager can’t simultaneously write code and manage.  As the team grows larger, additional layers of management should be brought in.  There are "phase changes" of sorts that an engineering organization (and the company as a whole) goes through when the total size hits certain ranges.  A CTO should be prepared to  help the company through these transitions.

So to summarize, the CTO may well start off as a strong individual contributor, but that function should gradually be reduced as the team grows in size. With that said, in order to maintain his or her edge and to retain “street credibility”, I believe a CTO must continue to develop and write code, as well as facilitate the architecture, and mentor others on the engineering team.


Most of the CxO’s on the executive team must play both an internal‐facing leadership role, and an external‐facing role facilitating investment and sales. The CTO is no exception. Initially as the company is getting going, probably more emphasis should be placed on the internal role. As the company grows, the CTO should transition to a having a larger external impact, through multiple channels. The CTO is often brought along as a “big gun” for important sales, in order to talk tech with the customer’s tech people. The CTO is usually part of the investor pitch team as well, as investors need to know that the company has what it takes to deliver the goods. In addition, the CTO should develop an external reputation for technical excellence and thought leadership, in the industry and his CTO peers in other industries. This is accomplished through several mechanisms such as blogging, articles and press releases, and talking at industry events.


I’ll close by saying that a major challenge of being a top CTO, is maintaining that technical edge that allows you to be a “player” in the player / coach model, yet spend enough time on the external facing role and continue being a thought leader. You can’t do this by just coding a few hours every couple of weeks. You need to sink your teeth into some problems.

Sunday, November 7, 2010

Passion, Joy, and Software Startups

Passion and joy are two of the most important parts of life, so as a software developer I think it's important to understand how to create a software career that brings us the most passion and joy.  As this eHow article on How to Follow Your Passion says:
Remember that following your passion is the most important thing you can do for yourself when it comes to your career. The majority of people are programed to not follow their passion when it comes to their career because they still fear the unknown. But following your passion early on is a must if you want to live a succesful fulfilling lifestyle.
One of the best things about America is that our culture supports entrepreneurs who build their careers by following their passion.  Although there is an interesting counter-point to this: The average Peruvian is 3.5 times more likely to start their own small business than the average American.  Maybe I should start my next startup in Peru.

According to the respected publication The Economist, between 1996 and 2004 America created 550,000 small businesses every month.  Wow!  And one of the best things about being a software guy in the Bay Area of California where I live, is that we are home to so many software startups.  In the SOMA area of San Francisco where I've worked at several startups, the number of startups has just started surging in 2010, according to The Wall Street Journal.  As SFGate puts it:
In San Francisco's South of Market neighborhood, Internet-based software startups are thriving despite the worst downturn to hit California since the Great Depression.
Some of you at this point will be saying, "but how many of those startups survive?"  The table at right shows the percentage of startups still in business up to 10 years later.  This data is from the Bureau of the Census produced for the Office of Advocacy of the U.S. Small Business Administration.  Another data point comes from Scott Shane's excellent book, The Illusions of Entrepreneurship.  In it, he states that one third of startups are successful after seven years, which agrees pretty strongly with the U.S. census data.  Scott characterizes this as "only one third", but I think of it like, "Holy Shit!  That's good odds.  Just have to try a few times and I'll get it.  Might even get lucky and hit my first one out of the park."

I believe following your passion is part of American culture and especially Bay Area culture, and is part of self-actualization.  This is pretty deep spiritual and philosophical territory.  Earlier this year I blogged over on my personal blog about the relationship between self-actualization and destiny.  Interestingly, Scott Shane states that the main reason people start a small business is to avoid working for others.  I think that perhaps it's harder to self-actualize if you're following someone else's business.

When you reap the reward of joy from pursuing your passion, you may also accumulate some amount of wealth - perhaps even a lot of wealth.  It's easy to forget that the value of following your passion is the joy you reap from doing just that - not so much from the wealth you accumulate.  In fact, one famous study suggests that lottery victims experienced less overall happiness than accident victims.  I take that to mean that wealth in and of itself doesn't bring joy or happiness one bit.  It comes from following your passion.  I love this quote from Health Guidance in the article True Wealth Will Make You Happy But It Must Be "True":
True wealth lies in defining ourselves by who and what we are, not by what we do or do not have.
Many software startups that started with lots of passion become successful, so they grow big and then they have a problem as they grow larger: Innovation and the entrepreneurial spirit flicker out.  How do they keep the innovation and entrepreneurial spirit alive as they grow?  Google has made famous (but did not invent) the idea of reserving a certain percentage of everyone's time for innovation - their famous 20 percent time.  Google says this in their section on an engineer's life at Google:
We offer our engineers “20-percent time” so that they’re free to work on what they’re really passionate about. Google SuggestAdSense for Content, and Orkut are among the many products of this perk.
I think the key part here is giving people a way to "work on what they're really passionate about".  By implication, the other stuff they're doing, well - they're not so passionate about that.  Hmm... That kinda sucks then, doesn't it?  That means 80% of your time is spent doing relatively boring stuff, and that to me seems like a high tax.  I'd kinda like it if I spent much more of my time working on stuff I'm passionate about.  Where can I do that?  And the fine print of Google's employment contract says they own 100% of any vision I create in my 20% time, so I'd like to keep at least some of that for myself please.  Where can I do that?

I believe there are at least two routes to finding more passion.  One route is to find someone else with passion that's out there building something, drink from their cup of passion (which should be overflowing), and then you will join with others in this way.  If you're following this route, I think it's important to find a group of people with passion for what they're doing - don't settle for people that don't care so much about what they're building.  If that kool-aid cup contains apathy, you will most likely have a bitter taste in your mouth.

Another route to passion is to realize your own vision in the form of a startup business, based on an idea you're truly passionate about.  How do you know if you're passionate about something?  I like the way Ryan Hoback puts it in his posting The Passion Behind Business:
When I get excited about a new venture that I have been developing, I feel the excitement run deep through my body. A feeling of invigoration comes across me. When a new idea comes to my mind, I begin to get happy like a little child opening a present on their birthday. You see, this is passion, and this is what successful business organizations are built on. Passion. Passion is the driving force and inner desire among entrepreneurs and leaders across the world which makes you want to continually pursue going above and beyond.
 Cash Miller puts it this way in his article Why is Passion in Small Business Necessary?
The number one asset a small business owner and entrepreneur needs to have is passion. Passion for your business is an essential ingredient when building a business. It can help to energize you when times are tough and it can help fire up your employees. Passion is what makes people believe. Do you have that passion?
So maybe you do have all the passion in the world, but most likely you don't have the personal resources required to start your own business all by yourself.  It takes a lot of time, which means you can't use that time to earn income, which means you must have a means of sustaining yourself while you're building your business.You can't eat passion for breakfast.

In addition to your own time, you must invest money in one way or another, either by eating into savings, or in the form of opportunity cost - lost opportunities for earning income, or in the form of loans, or an angel investor.

In addition to investing time and cash, if you're doing something really big, you will need to team up with other like-minded entrepreneurs as partners.  You will all need to drink the same passion, and it will take time for you to all get on the same page and share a passion for the same vision.

Tuesday, November 2, 2010

When not to use Agile

When building products with teams of people over the course of months, I think the Agile project management approach works best.  I have my own customized Agile approach I like to use that I'll likely describe in a future post.  But first, this posting describes when not to use Agile.

I've been noticing an increase in the number of extremely short, tactical coding exercises to deliver urgently needed functionality within hours or days.  More and more, I see people doing things like: throwing together a web site, building a prototype, creating a custom ETL solution, creating a new kind of widget, fending off a hacker attack, or creating a new report.  Sometimes the code may only be used once, or it might be used for years.  Often times these coding exercises must be delivered as quickly as possible, and with no advance notice.

I've found the best way to deal with these situations is to immediately create a team composed of the people that need to work together to deliver this.  Have them meet and figure out who's going to do what and when, and then they go off and do it.  No sprint meetings, no burn down charts, no stand up meetings - the team just gets the job done.

The challenge here is that the set of people that are working on these types of urgent tasks may largely overlap with the set of  people working as part of an Agile team, and they will most likely be in the middle of developing work that they've committed to delivering by the end of the sprint.

I think there are a few solutions to this problem, each with pros and cons.

First, you can reserve a certain total percentage of your Agile team's velocity for urgent tasks - say 20%.  This basically comes off the top of the total team velocity that you start with at the beginning of a sprint planning session.  At the same time, you identify a few stretch tasks in the sprint - bonus features that are either in the next sprint or that would be really nice to have.  If you don't use up your 20% reserve, you can tackle some of these bonus features.  The down side of this approach is the extra cost for context-switching mid-sprint; it takes extra time to switch between a person's sprint tasks and their 20% urgent task.

Second, you can just make sure that these two sets of people don't overlap - reserve different people for the short, tactical projects and manage that pool of people differently (but also probably not Agile).  You can periodically swap people between these two sets for cross training purposes.  The down side is that this probably requires more total people, so it's much more expensive.


Wednesday, October 27, 2010

Sustainable Software Engineering

Something bad happens to many software development teams somewhere around the 4th or 5th year of life of their code base: Quality and productivity take a nose dive, following an ever increasing trend of problems.  I call this the "Developer Duldrums" (with apologies to Norton Juster) because it's a sad place to work.  It's insidious because the team is usually building software the same way it always has, yet they've lost their mojo.  Why?  The answer is that the code base can no longer sustain development.  Enter sustainable software engineering practices.

Sustainable software engineering consists of:
  1. Automated functional testing.  I like to bias most of my QA team to writing automated functional tests.
  2. True unit tests (not functional tests masquerading as unit tests)
  3. Keeping your code bug free right from the beginning
  4. Relentless refactoring
  5. A solid and always-evolving architecture that is well internalized by the team
  6. Properly componentizing (with tests on the interfaces) and isolation of subsystems in your architecture so that you can re-write sub-components when needed without touching other parts of your code base
  7. Little overtime
Sustainable software engineering is usually not practiced because there is a cost associated with it, a cost which has a return that may be a year or more down the road.  However, if you haven't paid your dues and you get yourself a few years down the road, the chickens will come home to roost.  At that point, you're looking at rewriting your software - that's something you want to avoid because of the large cost and risks involved.


Counterpoint: In my experience, I've observed that technology trends move fast enough that approximately every five years, there may well be sufficient benefit to re-architect your code on top of newer technologies to justify a return on the investment required to do so.


So it may happen by luck that the Developer Duldrums curve may drop off close to the time you'd want to rewrite your product anyway because of advances in technology, so sometimes that's your way out of the Duldrums.  You kill two birds with one stone.  However, I wouldn't want to rely on such a coincidence happening.


Shorter projects benefit less from sustainable engineering, simply because you don't make your return on the investment because you don't have software you need to sustain.  So the amount of sustainable engineering you practice should be proportional to the project length or expected life time of the code base.


Sustainable software engineering isn't always possible to practice.  I would guess that there is approximately a 25% cost overhead for doing so.  This is due to the time it takes to write quality unit tests, refactor your code, and fix the important bugs as they occur.  (Counterpoint: The use of automated unit test generator tools such as Agitator from the beginning of the product development life cycle may reduce this tax.)  Often times, your primary objective is to reach your customers as quickly as possible to validate your market and product.  To do so, you could get to market faster by skimping on the sustainable engineering stuff.  That's certainly a valid argument.  But if you go down that route, you should be prepared for the chickens to come home to roost at some point.






Since I wrote this blog posting, I discovered the book Sustainable Software Development, by Kevin Tate.  I asked Kevin what he thought of my blog posting.  In his reply, which you can read below, he mentions something really interesting about proper componentization (which resulted in my point #6 above - thanks Kevin!):

I like the post and agree with the main points.  One thought is that you mention a 25% overhead, but in my experience people need to recognize that you typically get that 25% back (and more) through the knock-on effects of, for example, having reduced QA / manual testing and decreased effort behind release cycles.  
Another thought I had is that you touched on the need to periodically rewrite based on the latest technology.  That's where you might want to add one other "critical element" to your list: the need to componentize (with tests on the interfaces).  I talk about it indirectly in my book, but since I've written the book it's become increasingly obvious to me that this is another element of "secret sauce" because it allows teams to selectively rewrite their product as they go without having to throw out the entire system.  The rewrite from scratch scenario doesn't succeed very often, if indeed you even get the business support to do it!

Building high quality software fast and cheap

James Currier at Ooga Labs posted a blog entry that starts as follows:

"I’ve heard people tell me “We can build product fast, good, or cheap.  You can’t have all three.  Pick two.”  I believe this is a corrosive mindset, used by bureaucrats to justify mediocrity, or used by people who are afraid of failure to set the bar low enough so they feel comfortable in their daily lives."
I disagree with James and think you can only lock down two on a large, complex software development project.  While it's possible to build good software fast and cheap, I think that happens only under certain circumstances where the engineering risk is inherently low.  Also, sometimes you get lucky and nail it - I've seen that happens a few times. Projects having any of these characteristics generally can be delivered fast, good, and cheap:

1.  Young code base (generally a few months of development or less)
2.  Small team size (generally 1 or 2 senior developers, sometimes up to 3 or 4 if things are going well)
3.  Relatively simple feature set

As you get to bigger teams with more complex software that takes longer to create, your risk goes up.

The underlying principle at work here is "shit happens" - sometimes an engineer quits or gets sick, or you discover a fatal flaw in a library you're relying on and you need to retool, or you discover that a feature doesn't work as designed and you need to redesign it, or the market changes and you have to react.  When such a thing happens, the bottom line is that it's usually going to take you more time to do what you wanted to do.

So let's say such a thing happens AND you hold all three values fixed.  You can deal with such unexpected events by building in a "slop factor" to your schedule, so you eat into your slop time.  But essentially what a slop factor does is pull in the release date by saying "we expect this much shit could happen".  So you really are slipping in a way so the "fast" part is actually varying.  Or you could ask your team for overtime - that's one option.  But that's really tweaking the "cheap" part because you are putting more time in - if you paid for that hourly, your cheap just went up.  You could shave your feature set and simplify things and still ship, but then you're tweaking the "good" part.

Sometimes bigger shit happens than you budgeted for with your slop factor- we all get constipated from time to time - and if that happens, then what?  It's naive to believe that big shit won't ever happen.  Then what, if you hold all three fixed?

I call these three values by different names that more closely connote the values being managed:

1.  Release date
2.  Features & Quality
3.  Resources (people and hardware/software)

My basic form of project risk management is to look at these three values as dials that you can twiddle or tweak at the beginning of each sprint.  So I've folded this form of risk management into the way I practice Agile project management, by constantly reassessing the feature set, team velocity and composition, how the product is shaping up, and the importance of the release date relative to other business priorities, which may be more fluid than we'd like.

Wednesday, September 22, 2010

Why perform technical due diligence prior to an acquisition?

Michael Sisco put it well I think: "Conducting thorough [technical] due diligence can make the difference between a successful merger and a company's collapse."


If you're a high tech company looking to acquire another high tech company, please perform technical due diligence as part of negotiating the deal, ideally prior to signing a term sheet.  Here's what could happen if you don't.

First, you won't understand the costs, timeline, and resources required for delivering the promised integration of the newly acquired company.


Second, you won't have a good understanding of who to keep and who (if anyone) to let go as part of a Reduction In Force (RIF).  You won't be able to develop a new organizational structure because you haven't worked out what you're going to do and in what order.


Third, there is some value in the technology of the acquired company.  The value is significantly correlated with the quality of the code and architecture, and whether you have to rewrite the product code or not.  This should figure into the overall acquisition cost negotiation.


Fourth, you won't be able to optimize your own product development schedules to account for the possibility of the coming acquisition.  It's often possible to re-order development tasks to intelligently account for the possibility of the coming acquisition, without incurring cost if the acquisition falls apart.  Or you might pause or slow-roll hiring certain roles until you know how things are going to shake out.


There is always time to perform technical due diligence.  More information is always helpful.  For smaller companies, you can usually do the whole thing in a week.  Get the book Technology Due Diligence if you don't know how and want to do it yourself.  If you don't have the time or skills to do it yourself, you can hire a firm like blackduck to do the technical due diligence for you.


Since I manage the Chief Technology Officer Exchange over on LinkedIn, I asked some CTOs what they thought of the practice of investing in or acquiring other companies without performing technical due diligence.  Here is one response I find particularly interesting from a CEO in the group:
This kind of acquisition is a lot more common than it used to be. I'm seeing a growing number of companies decide they don't have to worry too much about moving software forward from its current state. That leads them to conclude they can do all sorts of things that seem like bad practices. Acquiring without diligence is one. Not taking care of the developer team is another. 
I haven't seen much evidence that these formulas are successful, it's just that I'm seeing companies try them more often. 
Another surprising one is for VC's to invest without proper technical diligence. This happens pretty often and has been going on a long time. They get stung by it periodically.
In general, most non-developer types have no real idea how to manage, value, or evaluate IP assets.