In this article, we'll discuss the results of Bitovi's checklist experiment and what key lessons we learned about making projects successful.

This is the last in a three part series of articles all about using checklists in your projects: why its important, creating a checklist you can start using, and what lessons we learned from our checklist.

We'll cover:

• Analysis of Bitovi's 9 years of project data.
• The surprising lessons our data taught us about what made projects succeed.

The Results

After filling out the checklist in hindsight for all of our previous projects, we calculated influence factors.

• Anything -1 to 0 is negatively correlated with success.
• 0 indicates no correlation with success.
• 0 to 1 indicates a positive correlation with success. The higher the number, the stronger the correlation.

Great Development

There were several key factors that correlated with success that were not a surprise.

• Do you code review everything: 0.28

Code reviews spread knowledge and make you more likely to write quality code simply because you know someone else will see it.

• Every module is documented: 0.28
• Every module is tested: 0.20

Projects that are well tested and documented are more efficient, cause fewer regressions, and have a lower total cost of ownership, so no surprise there.

• There is a continuous integration environment: 0.20
• There are separate dev, test, staging, prod environments: 0.20

CI and separate environments are indicators that the team is mature, and has processes in place to catch mistakes early, so they are correlated with success.

Some checklist items (the no-brainers) are there more to serve as reminders rather than create useful correlation data.

• Source control is used: 0.0
• Issue tracker is used: 0.0

All of our projects use source control and issue trackers, so there is no useful correlation data, but that doesn't mean it isn't a factor in success.

Great Design

The major surprise in our data was that development factors are not nearly as meaningful in predicting success as design and management factors.

As developers, this is not what we want to hear. We want to think that if we do everything right and produce the best possible application, we'll propel our project towards victory.

The most highly correlated factors in design were:

• User testing is performed: 0.45
• Design documentation (of some type) is created: 0.44

These values are nearly double the influence factors for dev. These particular factors are good indicators that your design team is plugged into what users want and that they are communicating decisions to managers and developers, which will streamline development.

Great Management

Even more than design, management factors played the largest role in predicting success.

• There is a Vision, Goals, Strategy, and Roadmap: 0.45

A clearly defined set of vision, goals, and strategy likely helps the entire team align, leading to clear, efficient decision making at every level.

• There are Yearly Trainings: 0.46

The yearly trainings themselves are probably not what is leading to success, but the fact that a company provides yearly trainings is another indicator that they have their priorities straight. It means the management encourages a culture of long term learning and growth. These types of managers are less likely to crack the whip unfairly and burn their team out, and are therefore more likely to succeed in the long run.

• There is a release in less than 6 months: 0.48

This was the single biggest predictor for success, probably because the longer a project goes on without a release, the more likely it is too complex. Complex products aren't typically as useful for users, run out of budget faster, and are more difficult to build. Release early and often.

• There are Social Events: 0.45

Valuing social interactions means bonding, a spirit of cooperation, and fun. We should not underestimate the value of team cohesiveness on the ability to solve problems together. Teams that have fun together probably communicate better and feel more free to take risks.

The Lessons

Here's what we hope you take from this article.

1. Focus on the real problems (which often aren't technical)

As developers, we get caught up arguing, or even obsessing, about technology choices. Which MVC framework should we use? Which module loader is the best fit?

More often than not, that’s not the problem, and worse, it's a red herring that distracts our focus and energy from what really matters.

Often, technology debates are like fixing a paint chip on the Titanic as it heads full speed towards an iceberg.

If we really care about our projects, we have to have hard conversations about things we weren't hired for.

This checklist has helped tremendously to organize and guide those conversations.

2. Use a Checklist

A simple checklist can be a fantastic tool for having those conversations with management without being confrontational. We encourage you to start going through Bitovi's JavaScript checklist or your own version of it with your manager and pointing out areas where your team could focus on improvement.

A checklist provides assurance that best practices and careful standards of quality are applied to every project.

3. Breadth over depth

The checklist data has taught us that it’s more important to scan for areas that haven't received any attention rather than perfect an area that has already received plenty of attention.

For example, rather than increasing test coverage from 70% to 80%, it’s much more valuable to add an automatic deployment script, or even better, to push for user testing.

We can have the perfect, most modern stack of technology and 120% test coverage, but none of that matters if the design doesn't solve the problem or the management doesn't have a clear vision.

At Bitovi, we've come to realize that we are brought in as technology specialists, but we care more about seeing projects succeed than winning a debate about client side module loaders. We assess our projects and pick our battles wisely.

If you truly care about project success, you have to care about more than just the code and development tools, and often make improvements or suggestions in the design and management of projects.

Keep reading

Why use a checklist?

Part 1 in this series, "Why You Need a JavaScript Project Checklist", talks about the power of checklists to transform industries.

What is in our checklist and how did we create it?

Part 2 in this series, "Creating A JavaScript Project Checklist", provides an overview of our simple JavaScript project checklist that you can use in your own projects.

When we applied this checklist to our previous 9 years of projects at Bitovi, surprising lessons did we learn about how to make a project succeed?

Part 3 of this series, "Your JavaScript Framework Doesn’t Matter – The Real Reasons Why Projects Fail", interprets the results from Bitovi's checklist experiment and highlights the key lessons we learned about making projects successful.