Unit testing has, during recent years, become a cornerstone practice in writing quality applications. Extreme programming advocates in the Java camp promoted use of unit tests, and their open source tool of choice, JUnit, quickly found C# fans via a version known as NUnit.
When Microsoft rolled out its new Visual Studio Team System, or VSTS, the company included support for unit testing. Microsoft did not support NUnit test data handling out of the box, however. To the consternation of some developers, such test data is handled by VSTS only via third-party plug-ins.
In unit testing, developers test individual bits of code before assembling them into a finished application. While unit testing may be on the upswing, some experts suggest its use is narrower than some suspect. As Will Ballard, owner of Mailframe, which makes the TestRunner tool for Visual Studio, noted, "Under 20% of developers generally unit test as a regular practice, and I still meet folks [unversed in unit testing] that I teach testing for the first time." A 2006 survey of TheServerSide.NET readers suggest a much higher prevalence of unit testing:
Which types of testing are supported as part of your software development process? (Choose all that apply.)
|Build verification testing||47.3%|
|None of the above||7.1%|
(Source: TheServerSide.NET Reader Survey, 2006)
Perhaps sleepy backwaters, where unit testing is the exception, will change with the new version of VSTS. It heralds a number of capabilities to support unit testing in an integrated environment. In theory, this will help manage the testing process so that testing a quality data can be shared and utilized by programmers, Quality Assurance QA professionals, and management.
Perhaps sleepy backwaters will remain untouched. To the extent that VSTS is aimed at the enterprise development shops, and is a high-end tool set, it may find first use among the teams that are already more likely to be on the testing bandwagon.
Soma speaks, er, blogs
More coherent test and development architecture will improve the process for creating high-quality, scalable software, some argue. In his popular blog, Soma Somasegar, corporate vice president, Developer Division at Microsoft, said "the Visual Studio 2005 Team Test Edition equips customers with tools to manage a wide array of unit tests, manual tests and web tests."
"Unit Tests work on functions and methods, and Visual Studio 2005 Team System makes it easy to manage and run unit tests," wrote Somasegar.
Somasegar and others point to some basic problems most people face with unit testing -- test creation, test organization and result reporting are key among these. Visual Studio 2005 Team System has implemented features to address these problems, Somasegar said.
The software lets users simply create tests. Test results are easily published to the Team Foundation Server. This enables developers "to share results via a portal site," he noted. "Visual Studio 2005 Team System treats testing as a top-tier activity helping reduce the risks inherent with delivering complex applications."
In addressing testing needs, Microsoft introduced unit-testing capabilities in the higher-end version of Visual Studio Team Systems, but not in the lower-end versions. Many developers have been up in arms about this lack of perceived core functionality.
As one developer, Alex Kazovic, posted in response to Somasegar's blog: "The problem is that unit testing is only available with Team System. Which customers are you looking to sell VS Professional to? Do you realize that these customers also want to do unit testing...Microsoft seems to be chasing the enterprise customer in terms of requirements and pricing and leaving behind its core customer base which will result in declining revenues and unhappy customers."
Mailframe's Ballard contended: "Putting unit testing in VSTS puts the price out of the reach of essentially the entire market except Fortune 500 shops. For me personally, in the short term, this leaves a pretty large market gap between the multiple thousand dollar price tag for VSTS and the $49 price tag for TestRunner, especially for folks that really just want unit testing and not all the VSTS feature set."
Eventually these capabilities could be incorporated into other versions of VSTS. Somasegar wrote, "In v1 (VS2005), these tools are going to be available only on VSTS Developer Edition and VSTS Suite. Over time, we will need to figure out which tools (and when) we can push down the VS stack. We won't get there for v1."
At launch, VSTS had three SKUs pieces for development, testing and architecture that plug into Team Foundation Server, which came online this year. Recently Microsoft added a database development SKU to the mix.
Ian Knox, lead product manager, Microsoft, said that, historically, testing is organized in terms of what developers and quality assurance roles do. But lately with the move towards agile concepts of development, more developers are doing some of the testing that QA people did in the past. A key aspect of this is the use of unit testing to ensure that chunks of code work, before they are plugged into the program as a whole.
VSTS supports five key types of tests:
- Unit testing, in which you call a class and verify that it is behaving as expected
- Manual testing
- Generic testing that uses an existing test application that run as part of the biggest test
- Web testing to ensure the html apps function correctly
- Load testing to ensure the app is scalable.
According to Knox, the challenge with the traditional testing workflow is that it is not an integrated experience. The developer checks in code and passes it off to the tester using third-party tools. When something does not work, it is entered into a defect-tracking database. VSTS improves the workflow by allowing the developers and testers to store their tests and results in one place. This allows a project manager to run a report to see how many tests have passed or failed and determine where the problems are, because the information is all in one data warehouse.
It's early to judge VSTS as a group development or testing platform. But some of Microsoft's common company traits may be coming into play. Chris Menegay, VSTS trainer/consultant with Notion Solutions, noted that Microsoft's test capabilities are not necessarily better than others; they just took a lot of existing concepts and made them work really well together.
Jay Flowers, a lead developer at Northrop Grumman, said, "One does not purchase VSTS for a unit testing framework or for a source control repository. It is purchased because it is an integrated platform to help enforce and encourage your chosen process. The main thing that VSTS offers over the other frameworks available is integration into the software development process."