Outsourcing test process

Development market in Poland become more adult during last couple of years and I am happy to see that there appears head hunting for new job titles like Quality Assurance specialists. That is a strong signal that software houses become more responsible for their products and their do understand that you can not leave the client alone with the software testing. The problem is that sometimes, because of different reasons there is no possibility to build such a team in-house. For example executives believe it is too expensive and despite your error cost analysis their refuse to hire quality specialist. The different reason may be that there is just one big project, which has releases once per month or three and there is no sense to hire full-time employee. These situations happen more often in smaller companies, however there are also situations in bigger once where developer or consultants are testing software and not regular quality specialists. The other topic is that there is quite many tools, which may help you during testing and you must be very experienced person to not buy a crap for big sum of money. There is many potential black holes like test automation, where you can waiste plenty of time and money without a significant effect. Where to install this tool? How to establish test environment which is totally separated from development and production systems (e.g. different domain)? Yes... there is plenty of question and things to do and most of developers don't want and should not make their hands dirty in this mud. Besides... lets be honest. Testing is an adventure and there is often that is repetitive, benedictine work. This is why outsouring QA work is quite interesting solution. The question is only how to do it smart, because it is very easy to get pile of wastful papers from the external company, which does not give any add-on values to your development process.

1. Which cash flow model to choose?

There are generally following possibile models:

A. Times and material - where you pay for time, which people from the external company spend on your projects. There may be different hours wage for people who prepare tests and write final tests and people who actually do the tests. That is definitely the best option where you start cooperation with external company as both sides can easily switch the level of cooperation and adjust it to real needs.

B. Fixed price - if you have some bigger project(s) and there is possiblity to define easily the range of cooperation plus you want to transfer some part of responsiblity on external company, you may think about preparing the contract. The problem is that most usually the border between develoment and testing is quite blurred and this type of deal has always the risk of doing 'only what is in contract'. As live brings many surprised you may find yourself easily in unpredicted situation, where you will have to come to A option any way (e.g. testers must explain developers what the problem is about and contract does not predict such option.

2. What is needed to prepare cost estimation?
That is probably the hardest topic, when you start coopeartion. It is smaller problem in option A as both sides realize that the total value is flexible, but it is very hard topic in option B and most probably outsourcing company will add there some reserver/risk budget for unpredictable situations. Anyway... There must appear answers on number of questions...

A. What is the size of work? The best way is a simple presentation of application and discussion which parts of the system will be tested. Then, basing on number of screens/forms or number of functions plus predicting the number of test iterations, external company will have to estimate their engagment.

B. Who maitain and provide externally test environment?

C. Who installs the software in test environment?

D. What type of tests are done? You have wide range of options:

• code review - basing on provided, general or existing rules you provide report about the code purity
• unit tests - writing unit tests for code (usually should be done by development team)
• integration - putting all the modules together (usually should be done by development team and good idea is to link them with building process - smoke test with daily builds is here a good option)
• installation - test of installation package in different environments
• functional - testing all the functions of code (the minimum package is called acceptance tests)
• platform - testin application on different type of machines, OS systems, internet browsers etc.
• performance - checking how the system behaves (eg. what is response time, how much RAM, CPU, disk space is used) during heavy workload (e.g. many users at one time).
• security - checking if the system is secure and nobody from outside or inside can break in, there is also the option of doing the hardening
• destruction - emulating dummy users (eg. monkey test - play on keyboard like the monkey)
• documentation - check if you can use the system without any interaction with software vendor
• legal - verifing if the application follows particular legal regulations

...

E. What is test approach? Generally the tests are done basing on prepared test scenarios (accepted by both sides) plus some number of ad-hoc test. The question is how many of tests are basing on written scenarios and how many on intuition?

F. What is communication model?

INPUT: The best option is if the external company gets the packed product with instructions and they should do all the tests alone. Then they emulate the real users and situation is quite clear.

OUTPUT: By default the external company will provide the summary report and some list of errors. Most important thing is to have full and detail reporduction list in case of any bug. It should contain the number of action from particular, written test scenario if possible plus as many screenshoots as possible.

Roadblocks
It happens quite often, especially an early stage that there happens the road block - the event which stops QA team from testing all application, module or particular, bigger set of functions. It is opened question how both sides deal with this option and there is wide range of option starting from the very formal one where external company closes whole iteration of tests offically as it is to the very informal where developers delivers quickly some patches. Both solutions have though positives and negatives and each case is different.

Bug Tracking System
More adult organization will provide Bug Tracking Systems, which allows to monitor the life cycle of errors. The open question is who maintain the system and what are the rules of coopeartion? Choosing the particular platform is quite easy thing as there is plenty of options - there is number of interesting web-based open source solutions.



Oursourcing testing process is quite challenging task and there is some risk of miscommunication as testers are usually adding the work to developers and developers are writting buggy code ;) If both teams are located in different companies managers must put the real effort in communication improvment and perfecting the process. On the other hand the external company gives a fresh look at the appliaction, bigger independence in testing and kind of the insurance that a final product is usable. Except detecting errors they may also add number of suggestions/change requests, which make the system *turely* more userfriendly and smooth.