Bliki: VcsSurvey

Bliki: VcsSurvey: "

When I discussed VersionControlTools I said that it
was an unscientific agglomeration of opinion. As I was doing it I
realized that I could add some spurious but mesmerizing numbers to
my analysis by doing a survey. Google's spreadsheet makes the
mechanics of conducting a survey really simple, so I couldn't
resist.

I conducted the survey from February 23 2010 until March 3 2010
on the ThoughtWorks software development mailing list. I got 99
replies. In the survey I asked everyone to rate a number of version
control tools using the following options:

• Best in Class: Either the best VCS or equal best
• OK: Not the best, but you're OK with it.
• Problematic: You would argue that the team really ought to be using something else
• Dangerous: This tool is really bad and ThoughtWorks should press hard to have it changed
• No opinion: You haven't used it

The results were this:

Tool	Best	OK	Problematic	Dangerous	No Opinion	Active Responses	Approval %
Subversion	20	72	6	1	0	99	93%
git	65	19	1	0	14	85	99%
Mercurial	33	27	2	0	36	62	97%
ClearCase	0	3	14	41	41	58	5%
TFS	0	0	32	22	44	54	0%
CVS	0	14	59	11	15	84	17%
Bazaar	1	13	3	0	80	17	82%
Perforce	1	26	16	1	54	44	61%
VSS	1	1	11	64	22	77	3%

As well as the raw summary values, I've added two calculated
columns here to help summarize the results.

• Active Responses: The total of responses excluding 'No
  Opinion'. (eg for git: 65 + 19 + 1 + 0)
• Approval %: The sum of best and ok responses divided by active
  responses, expressed as a percentage. (eg for git: (65 + 19) / 85)

The graph shows a scatter plot of approval percentage and active
responses. As you can see there's a clear cluster around Subversion,
git, and Mercurial with high approval and a large amount of
responses. It's also clear that there's a big divide in approval between those
three, together with Bazaar and Perforce, versus the rest.

Although the graph captures the headline information well, there's
a couple of other subtleties I should mention.

• Although the trio of Subversion, git, and Mercurial cluster close
  together on approval, git does get a notably higher amount of best
  scores: (65 versus 20 and 33).
• VSS got the most 'dangerous' responses, but a couple of people
  approved of it.
• Neither TFS or ClearCase are liked much, but ClearCase got more
  'dangerous' responses than TFS (41 versus 22).
• Don't read too much into small differences as I'm sure they aren't
  significant. I'm sure the difference in approval percentage between VSS, TFS,
  and ClearCase isn't signifcant, but the difference between these three
  and the leaders is.

Some caveats. This is a survey of opinion of ThoughtWorkers who
follow our internal software development discussion list, nothing more. It's
possible some of them may have been biased by my previous article
(although unlikely, since I've never managed to get my ThoughtBot
opinion-control software to work reliably). Opinions of tools are
often colored by processes that are more about the organization than
the tool itself. But despite these, I think it's an interesting data
point.

I should also stress the important point to take away from this
isn't the comparison between those close in the numbers, eg comparing
git and Mercurial or comparing TFS and ClearCase. Any survey like this
has a certain amount of noise in it, and I suspect the noise here is
greater than such a difference. The important point is the big
approval gap between the leading tools (Subversion, git, and
Mercurial) and the laggards - essentially the point in
VersionControlTools.

"