Pokazywanie postów oznaczonych etykietą quality assurance. Pokaż wszystkie posty
Pokazywanie postów oznaczonych etykietą quality assurance. Pokaż wszystkie posty

czwartek, 7 sierpnia 2008

Outsourcing test process

Development market in Poland become more adult during last couple of years and I am happy to see that there appears head hunting for new job titles like Quality Assurance specialists. That is a strong signal that software houses become more responsible for their products and their do understand that you can not leave the client alone with the software testing. The problem is that sometimes, because of different reasons there is no possibility to build such a team in-house. For example executives believe it is too expensive and despite your error cost analysis their refuse to hire quality specialist. The different reason may be that there is just one big project, which has releases once per month or three and there is no sense to hire full-time employee. These situations happen more often in smaller companies, however there are also situations in bigger once where developer or consultants are testing software and not regular quality specialists. The other topic is that there is quite many tools, which may help you during testing and you must be very experienced person to not buy a crap for big sum of money. There is many potential black holes like test automation, where you can waiste plenty of time and money without a significant effect. Where to install this tool? How to establish test environment which is totally separated from development and production systems (e.g. different domain)? Yes... there is plenty of question and things to do and most of developers don't want and should not make their hands dirty in this mud. Besides... lets be honest. Testing is an adventure and there is often that is repetitive, benedictine work. This is why outsouring QA work is quite interesting solution. The question is only how to do it smart, because it is very easy to get pile of wastful papers from the external company, which does not give any add-on values to your development process.

1. Which cash flow model to choose?

There are generally following possibile models:

A. Times and material - where you pay for time, which people from the external company spend on your projects. There may be different hours wage for people who prepare tests and write final tests and people who actually do the tests. That is definitely the best option where you start cooperation with external company as both sides can easily switch the level of cooperation and adjust it to real needs.

B. Fixed price - if you have some bigger project(s) and there is possiblity to define easily the range of cooperation plus you want to transfer some part of responsiblity on external company, you may think about preparing the contract. The problem is that most usually the border between develoment and testing is quite blurred and this type of deal has always the risk of doing 'only what is in contract'. As live brings many surprised you may find yourself easily in unpredicted situation, where you will have to come to A option any way (e.g. testers must explain developers what the problem is about and contract does not predict such option.

2. What is needed to prepare cost estimation?
That is probably the hardest topic, when you start coopeartion. It is smaller problem in option A as both sides realize that the total value is flexible, but it is very hard topic in option B and most probably outsourcing company will add there some reserver/risk budget for unpredictable situations. Anyway... There must appear answers on number of questions...

A. What is the size of work? The best way is a simple presentation of application and discussion which parts of the system will be tested. Then, basing on number of screens/forms or number of functions plus predicting the number of test iterations, external company will have to estimate their engagment.

B. Who maitain and provide externally test environment?

C. Who installs the software in test environment?

D. What type of tests are done? You have wide range of options:
  • code review - basing on provided, general or existing rules you provide report about the code purity
  • unit tests - writing unit tests for code (usually should be done by development team)
  • integration - putting all the modules together (usually should be done by development team and good idea is to link them with building process - smoke test with daily builds is here a good option)
  • installation - test of installation package in different environments
  • functional - testing all the functions of code (the minimum package is called acceptance tests)
  • platform - testin application on different type of machines, OS systems, internet browsers etc.
  • performance - checking how the system behaves (eg. what is response time, how much RAM, CPU, disk space is used) during heavy workload (e.g. many users at one time).
  • security - checking if the system is secure and nobody from outside or inside can break in, there is also the option of doing the hardening
  • destruction - emulating dummy users (eg. monkey test - play on keyboard like the monkey)
  • documentation - check if you can use the system without any interaction with software vendor
  • legal - verifing if the application follows particular legal regulations
...

E. What is test approach? Generally the tests are done basing on prepared test scenarios (accepted by both sides) plus some number of ad-hoc test. The question is how many of tests are basing on written scenarios and how many on intuition?

F. What is communication model?

INPUT: The best option is if the external company gets the packed product with instructions and they should do all the tests alone. Then they emulate the real users and situation is quite clear.

OUTPUT: By default the external company will provide the summary report and some list of errors. Most important thing is to have full and detail reporduction list in case of any bug. It should contain the number of action from particular, written test scenario if possible plus as many screenshoots as possible.

Roadblocks
It happens quite often, especially an early stage that there happens the road block - the event which stops QA team from testing all application, module or particular, bigger set of functions. It is opened question how both sides deal with this option and there is wide range of option starting from the very formal one where external company closes whole iteration of tests offically as it is to the very informal where developers delivers quickly some patches. Both solutions have though positives and negatives and each case is different.

Bug Tracking System
More adult organization will provide Bug Tracking Systems, which allows to monitor the life cycle of errors. The open question is who maintain the system and what are the rules of coopeartion? Choosing the particular platform is quite easy thing as there is plenty of options - there is number of interesting web-based open source solutions.



Oursourcing testing process is quite challenging task and there is some risk of miscommunication as testers are usually adding the work to developers and developers are writting buggy code ;) If both teams are located in different companies managers must put the real effort in communication improvment and perfecting the process. On the other hand the external company gives a fresh look at the appliaction, bigger independence in testing and kind of the insurance that a final product is usable. Except detecting errors they may also add number of suggestions/change requests, which make the system *turely* more userfriendly and smooth.

czwartek, 15 listopada 2007

fxCop story

It is quite often, that companies posses their own “Rules of coding” document like the one from Philips, which is more or less the compilation of documents available on the net. I strongly suggest also reading "Framework Design Guidelines", which is my bible in the area. One thing is to write the paper and the other is to have it truly implemented. Some companies have the process established that none code may be checked into repository without the code review performed by the other developer, but that is open issue is if this process ensures that all rules are always followed and exceptions from obeying rules are registered. That is why, it is always nice to have the cyber cop, who will check once per a while if codes in your repository follows the rules established in written documents. The cop has even a name on this t-shirt and heeeeeaaaaar heeeeee issss - fxCop.

Each project is different and each time you must select, which rules will be used. I think that the best way to deal with the issue is the following:

1. Create fxCop project
Get all dlls and exes which will be analyzed and add them in target section. Analyze them and save the whole project.

2. Filter sets of rulesFor example, if you are not planning to localize your software you do not need Globalization Rules at all. In similar way, if you do not have the legacy problem, you probably do not need the Interoperability Rules.

3. Filter rules
Check each rule. More errors is in rule, in more details you need to analyze what exactly the rule is about. In some situation fxCop provides you the link directly, in other situations you must look at help or ask MSDN manually about the rule. You will find quickly that some rules are useless for you like (for me):

  • EnumsShouldHaveZeroValue – we use enums just internally and they are not public
  • AvoidNamespacesWithFewTypes – the project is at early stage and further extensions are planned shortly
  • MarkAssembliesWithClsCompliant – we do not need to be compliant with CLS

And some of them are golden one (for me):

  • IdentifiersShouldBeCasedCorrectly – priceless thing, which usually would take me a lot of time, if performed manually
  • AvoidUncalledPrivateCode – it helps me to find quickly unused code
  • DoNotCallPropertiesThatCloneValuesInLoops – it never should happen

Except checking MSDN, if that is necessary open the code and check where particular error happens in code. It will happen often that only looking at code, you can answer if the problem is dummy or real.

Analyze the code again and check how much work there is to standardize the code. If there is too much, than you can afford you need to repeat step 2 and 3. If you can, talk with your developers to ensure, that your point of view is valid.

4. Look at errors sets and ensure, that they are valid
Forget for a second about fxCop and how cool it is and try to put yourself in developers shoes. Are enlisted errors really important; do they really need to spend a time and risk the repair (which may cause additional errors)? Remember that you should implement the whole thing incrementally to convince people that it truly empowers they work. If you start with too many rules at the beginning, you will cause people to make refactoring for months, what will not make them and your boss happy ;) That is why, it is good to “thing big and start small”.

Filter the last time, the list of suggestions for developers.

5. Establish the process
The tests have been performed for the solution, which consists of 62 projects and has around 30k LOC.

Option A
Run the report from command line manually. The default xml report is parsed nicely by default by IE – options like expand/collapse are possible there without any configuration. You may generate the CSV in the same way, but in fact that is the same xml (just extension changes). The disadvantage is that you need to run the whole process manually and you need also to update the list of targets as the outputs (dis)appears, but the plus is that it does not disturb the daily work of developers. The batch like:

fxcopcmd /p:"..\..\..\_fxCop\ESPO1v0.FxCop" /c /o:".\FxCopReport.xml"

Took me about 1:21.

Option B
The other idea is to apply the fxCop process as after build for Release version of the product (“Adding FxCopCmd to Your Build Process”). In this way, daily Debug builds are quick and when everything is ready key developer does Release build and gets immediately the report of things to repair. Unfortunately post-build actions are available on the projects level and not on solution level, so you must either apply it to each project separately or once to project, which is build at last in the whole solution, if you are sure it will be always compiled as last one (Project Build Order). In this case, you must be also sure that each person:

  • Has fxCop installed
  • It is installed at the same place or path is set properly
  • fxCop project is downloaded into the same place at each PC (shared folder possible)

FxCopCmd /p:"D:\SVN\ESPO\fxCop\ESPO.FxCop" /c /o:"D:\SVN\ESPO\fxCop\FxCopReport.xml"

The other option suggested in here is to use the VS 2005 add-in and link it with MSBuild process.

Option C
Even more interesting option is to add the action as a post-build in any project. You need to change the fxCop project and clean Targets tab. You may establish that folder, where fxCop has been installed is added to the Path global variable and common folder with fxCop project and where all the reports will be stored. You need then to ensure that post-action is added to each project to have the whole thing working

Fxcopcmd /f:"$(TargetPath)" /p:"..\..\..\_fxCop\ESPO1v0.FxCop" /c /out:"..\..\..\_fxCop\$(ProjectName)-fxCopReport.xml"

You need to know, that my sample solution has compiled in 32-45 seconds without fxCop and in 8:10 with fxCop for option C (even when just 5 rule sets were turned on)! Only iIf you don’t have too many projects, it may be the good option. In other cases option A or B, will be probably better.

Whichever option you chose, there has to be some negotiation stage (especially at the beginning until the set of used rules will not stabilize), when developers may give the opinion about the usability of the particular rule or changes. When it is established and the set of rules is redefined, it must be double checked after some time if the errors are repaired

You may also consider at later stage to write your, customized rules, but you always must consider if it really safes time ;)

Conclusion
You must spend some real piece of time to make the whole thing to be REAL, but I have a strong believe that the automated effect is usually worth of it especially if you the project is:

  • long term
  • open source or you need to pass the rights to source codes somebody else
  • the code is analyzed externally
  • to which the others may write plug-ins
  • developers switches

Of course you must review the whole thing once per while and redefine the set of used rules (make the tolerance more narrow or wide), but the even the knowledge about the post-build process will motivate developers to write better code.

At least, but not at last that is important to mark, that automated process is just the support and not replacement for manual code reviews.

poniedziałek, 8 października 2007

5 best ways to have the house full of bugs

Mood: Just after sex
Music: Elektrobank

There was a question on the LinkedIn "What are the Top 5 reasons that NEW applications are released to production prematurely?" and just 4000 letters to answer it! Not enough... Full answer published below...

It is quite hard to discuss with such a guru like Joeal Spolsky, who answers perfectly your question directly in the article. Anyway the life is about the challenge, so lets try to watch fellows, who causes that software is immature. They say:

1. We do not need QA!

It sound crazy, but that is true and I met it couple of times. Many software houses do not have the regular QA team! It happens not only because “we can not afford” (the number 5 on Joel’s list), but also because the decision makers believe “we do not need it”! I met once a very clever director, in a mature consulting company, who believed (despite number of my suggestions), that consultants can perfectly find all the bugs, when preparing the demonstrations and installations!

If QA exists, very often these guys are responsible for the other duties like builds, version management or configuration. I know the companies, where one tester is controlling the work done by 15 developers! Joel Spolsky says about 1 tester per 2. Gurock goes even further writing about dedicated software - one per one. In my opinion well trained programmers, who does unit tests can have 1 tester per 3 programmers, but they definitely need QA team, who is focused on testing and nothing else.

2. QA is tiring and boring!

Many people feel it inside their hearts and they are not strong enough to admit it. We are amazed by the possibilities of new technologies and so tired because of countless number of errors. Who like the messenger, who brings the bad news? Who wants to hear about hundred of the very specific situations, when the system does not behave, as it should? Why do we need to repeat the same routine (test script) again and again and again? Ooo my goodness, do you report the same error again? The damn QA team… they are the pain in my ass!

This is why many errors are hidden under the carpet, some inconvenient scenarios are omitted and test reports are so optimistic. It is so rare to hear: "Yes, that is sometimes boring, but it is necessary to do and I will not release the product without having this check list working in 100%!". I promise my QA team nothing else than sweat, pain and… support.

3. I want just demo version!

Very often the sponsors of the projects want just to feed sales forces, so they can present the amazing, break-through functionalities on the conference. The first one, the second one, then there is a first client... lets just install it only for this customer. We will do the real re-factoring later after we cash this deal. The real horsepower is irrelevant until the regular users start to report zillion of bugs and then the witch haunt starts :)

The other variant of this story is “add this, this and this functionality, and keep the quality!”. Sometimes it is simply impossible to froze the scope of the application (especially when agile methodologies are used) and then the application may truly explode with number of functionalities :)

4. Automated, unit self-testing is the panacea!

Part of many agile methodologies is the automated unit testing. Each function should have the shadow function, which automatically tests its effect. The idea is truly great, widely popularized, but even when the company says loudly that they use XP or Scrum only some functions are truly automated. Furthermore, in the situation, it is quite easy to neglect the other type of tests like integration, platform, destruction etc. Do we need it, if programmers do the tests on the lowest level?

Yes, we need them! If you do not do all these things, they will happen sooner or later unless you bankrupt before the first commercial release :)

5. Do not write papers - just test it!

If the QA team is not instructed to document the scenarios, work done, errors – they will not do it! Most of people prefer to speak than to write, to keep the knowledge than to share it. The decision of the final release must be based on hard, written information like the number of bugs per test iteration (how close are we to Zero Bug Bounce ), level of code coverage by tests, performance and security test reports plus number of the others. Feelings are not enough in this business!

QA team is paid for being a couple of steps before any user and knowing the strangest possible way of using the software (Joel’s 3rd story). If the final user is finding some unknown bug, the QA team fails . Not all bugs must be fixed, but all of them must be documented! You can do it only if you keep some reasonable level of documentation – check lists (test scenarios), bugs registry with reproduction list and statistics.



At the end I want to express quite brutally that the most often the business decision makers are responsible for these things as they take care about quality only verbally! Their declarations are not followed by real actions and investments. The regular quality reviews are neglected as the new, cool functionality or the customer behind the door is more important. Most of them are not capable to stay with Sara Ford talking at channel 9 until she gets into the lab. Yes, she is not sexy and she is boring :)

ps.
She probably can not also dance as the one on the videoclip, but she as amazing as the one :)
 
web metrics