Testing tools have long been disparate, and often far removed from development processes. Bringing better integration to test and development is one of Visual Studio Team System's main goals. VSTS tries to make test and report correlation easier by providing a full-featured tool; yet it does support third-party automated test plug-ins for people who want to mix and match.
When Microsoft went about fashioning a new tool suite, integration was paramount. According to Sam Guckenheimer, group product planner, VSTS, Microsoft, research indicated that project managers spent 40 hours a week cutting and pasting. "We tried to take that down to maybe a couple," he said.
"Instead of having tools that worked in different ways, we said 'we will instrument all of the activities and put them into a multidimensional warehouse.' We have been doing this with supermarket data for years," joked Guckenheimer.
The move to integration and VSTS, which supports a Microsoft unit testing tool, is not without controversy, because many developers judged that NUnit and related tools served them well. With VSTS, testing capabilities have been integrated right into the VS environment, simplifying the process of writing tests without flipping back and forth between applications.
But some urge caution and recommend an effort to size up the problem space. They say, for small teams and some quick projects, VSTS may be seen as overkill. Said Yaron Sinai, CEO, Elementool, "Team system is a more complicated solution." In fact, Elementool's alternative is Web-based testing as a service. There is virtually no set-up stage.
"The way we see it, VSTS is for management of large projects. In order to use the system, you have to set up a server and install it and maintain access to all users, and if they are in remote locations, you need to open the firewall to access it. With Elementool," he compared, "you don't need to install anything."
Sinai noted that the latest version of Elementool's service application bug tracking tool, announced in June, deploys directly in the Microsoft Visual Studio 2005 development environment.
VSTS supports testing as a part of an automated build system. Said Chris Menegay, trainer/consultant with Notion Solutions, "Before, you had to hook NUnit into [the larger automated system] and figure what to do with test data from a build report standpoint. The more complicated the stuff you tried, the more complicated it got."
He continued: "All the tools did their niche things, and you had to bring each in, and create a workflow around them. You used to spend a lot of time building up the tooling."
Menegay favors the fact that, with VSTS, test results can post to a SQL Server database, which is something that did not occur with other tools, unless you wanted to build the software infrastructure.
VSTS also allows a developer to aggregate multiple tests, which have already been written, so that they can be executed by automated test agents to simulate up to a thousand users for load testing. Several agents can be run concurrently to increase the load in multiples of about a thousand. This whole process allows a team to reuse the work that was initiated for the various kinds of tests.
This also makes it easier for the developers themselves to execute a load test based on the unit tests of the individual code modules, so that they can identify problems at an earlier stage, saving time, and learning how to write better code. Can it prevent service related problems that had not been foreseen when the app was being developed? That is a goal. As Microsoft's Ian Knox noted, "The last thing you want is your pager going off at 3 a.m.."
Various automation capabilities also are said to ease the lot of developers who must create unit tests, simply by right clicking on methods. They can do things like add data collection points, export to XML and attach bug reports. Sonal Pardeshi, product manager for Visual Studio Team Test at Microsoft, distinguishes the new Microsoft offering, saying that with alternate approaches using multiple tools, it is often difficult to correlate tests with reports.
Menegay, of Notion Solutions, estimated automated test code generation capabilities can save a developer two to three minutes of work per method. As Menegay noted, "Writing six or seven lines of code is not a lot. But if you have 500 methods, you just saved 1,000 minutes of your life."
Unfortunately, this improvement will not be of much use to agile programmers who believe in writing the test before they code, since the automatic test generation is based on the method to be tested. But they will still get the savings of not having to go back and forth between tools, and there is a unit test wizard for those new to unit testing, Menegay explained.
Another key advantage, Menegay said, is the Test List feature for managing tests in hierarchical categories that can be shared across projects. Previously, a developer did not necessarily have a good user interface to manage tests on an ongoing basis. He said, "With NUnit, you would create the tests with projects and put category attributes in place, and you can still do that, but you can also take a higher level view and go across projects."
Now, there are two grouping mechanisms for the tests, using projects and folders, using Test Lists. Menegay said "Test Lists makes it easier to manage your tests in batches. If I have twenty tests in one project and twenty in another, I can include them by clicking on one group of text boxes and reorganize my groups very easily."
Of course, there is the creeping possibility VSTS and its new and enhanced capabilities will drag developers deeper into testing -- not necessarily a favorable outcome.
Sharing the load
Improving software -- not overburdening developers -- is the goal, indicated Compuware's Ken Cowan, product manager of DevPartner Product Management Technology at Compuware, whose company is among the prominent third-parties able now to extend VSTS.
"We are not advocating that developers replace QA [Quality Assurance] people," he said, "But it is good, if they do unit tests, that developers do tests from a user perspective. Now we attach the test data to work items, as assets they can share."
Compuware DevPartner and TestPartner provide interactive feedback, coding advice and corporate coding standards and enable developers to find and repair software problems. This software now plugs into VSTS to make detailed diagnostic data available to the developer.
Cowan foresees deeper integration in the future. "Team System is really compelling as collaboration vehicle," he said.
Among other third-parties supporting VSTS are AutomatedQA with TestComplete and Borland with CaliberRM. But as important as third-party vendors may be the open source community, particularly those aligned with NUnit.
Clearly, NUnit is not natively supported by VSTS. It is one of the most widely used pieces of open source software in the .NET world. Jim Newkirk, development lead for the Microsoft Platform Architecture Guidance team, recently created a program that converts some C# NUnit code into tests compatible with Visual Studio 2005 Team System. For now, Newkirk said, it is aimed at people who are "evaluating Visual Studio 2005 Team System but really don't want to convert their existing code by hand."