For the first time in my learning dump I would like to post how I did a case study in understanding Quality Assurance (QA) Practices in my current organization. Unlike, tech giants where software engineers do both development as well as testing my current org still follows DEV & QA teams hierarchy with developer to QA ratio on an average 5:1 ( there might be N arguments for or against any developers to QA ration let me not get into that.) All I wanted or what's expected out of me is to understand the current practices that's been followed in the organization. As a pretty new member of org it was a challenging to figure out all the QA folks and get some time from them (not all though) as I could see the a slender fear factor with lots of questions when I reached out (Back to the topic).
To understand QA practices first I need to know the list of areas that I need to cover before I could have the discussion with any of the QA engineers. The following are the two areas in which I wanted to scope my discussions:
- Testing Methods
- Testing Strategies
Testing Methods
In testing methods I had concentrated on how we are doing the testing. Initially I was in a tight corner whether should I really need to understand or even speak about Testing methods but then (after speaking with couple of folks), I got a firm belief that I should speak about the testing methodologies that's been followed in their respective projects. Areas that I covered in testing methods can be grouped into two as follows:
- Box Approach
- Static & Dynamic Approach
Box Approach
One might have a big question why would one be very much interested whether I'm doing a Black/White/Grey box testing. But, I wanted to understand whether QA folks has the real essence of what they have been testing. During my discussions all those who are into automation said they are into White box testing up front on further questions like "do we really go through the code that's been delivered by developers" then they have changed the mindset from White box to Grey box testing. Whereas in the case of functional test engineers they said it's black box and we are moving towards White box or automation. What I have understood from all is most of QA engineers has an assertion that when we do automation then we are doing white box testing.
Static & Dynamic Approach
There are more than few hundred tools that helps us to ensure whether we are delivering a quality product or not and each of these tools has it's own approach to identify the bugs. Some belong to static and the most belong to dynamic analysis. As we could see there is a predominant discrimination in number tools that's been available in market and I could see the same kind of discrimination in mentality of QA engineers as well, most of them are really focused on dynamic testing it's one of the area where we could stress up on so that we could avoid a significant number of hidden bugs in the code.
Testing Strategies
After having a brief discussion on testing methods my focus to understand QA practices was to dig further to understand all the strategies that's been followed. I had some areas in my mind which seems to be must cover to understand about testing, the following are group of areas that I wish to cover in my discussions:
- Test Levels
- Test Coverage
- Testing Tools
- Security Testing
- Load / Stress Testing
- Risks & Mitigation
- Test Schedule
- Regression Approach
- Test Status Collections & Reporting
- Test Records
- Requirements Traceability Matrix
- Test Summary
Test Levels
In the test levels I wanna see how we are actually engaging with our testing in the following levels:
The reason why I wish to touch upon Unit Testing as most of us were practices Grey Box or Black Box testing. With no surprise all QA folks had their concentration on other levels of testing and they had a firm belief that UT's are to be handled by developers and QA does not need to engage in UT whereas, my intention was to see if QA are really aware of Code Coverage of the application through UT (Will cover more about Code Coverage in a little while).
Functional Testing
Functional testing is the major level of testing that's been concentrated by all the QA folks across the org. The method or the way in which testing is carried out has been covered in Testing Methods i.e. in the first few paragraphs of this blog.
Integration Testing
Integration testing can be viewed in two aspects the first one deals with the upstream systems and the second one deal with the downstream systems. Most of the time upstream system integration's issues or bugs are uncovered in functional testing. Whereas, in the case of downstream systems we might not be aware of the consequence of updating the existing flows. There is a common view among most of the Quality Assurance folks that it's the responsibility of downstream system team to take care of integration issues. Ideally, it's the responsibility of upstream system to make sure that there is no issues while upgrading the flows or we should inform them regarding expected outage or breakage when there is a change.
Test Coverage
Test coverage is one of the metric which has been perceived by QA folks as an area that needs to be covered by developers with unit tests. In reality the metric has to be maintained by QA team that includes dynamic coverage as well not much are really aware of how get the code coverage in dynamically and this is the area that needs a strong awareness.
Testing Tools
There is new testing tool that might be built or released by the time you are reading this blog but, how many ever tools that's been rolled out we should do a legitimate study or analysis before we pick up them in the stack of tools or technologies that's been in use already.
Security Testing
Not much of QA folks are aware of tools that help with testing security loop holes or the kind of security issues that might pop-up in the projects. I firmly believe that every one in the organization should have a certain knowledge on security testing practices.
Load Testing / Stress Testing
A Testing area which is also seen as a specialist job. And there are few so called specialist have presumption that the load generated is directly proportional to the number users that's been configured or used in the settings.
Requirement Traceability Matrix
Test Schedule
Test Records
Test Summary
Risk & Mitigations
All above areas are getting endangered as we have moved to so called agile life cycle.
In the test levels I wanna see how we are actually engaging with our testing in the following levels:
- Unit Testing
- Functional Testing
- Integration Testing
The reason why I wish to touch upon Unit Testing as most of us were practices Grey Box or Black Box testing. With no surprise all QA folks had their concentration on other levels of testing and they had a firm belief that UT's are to be handled by developers and QA does not need to engage in UT whereas, my intention was to see if QA are really aware of Code Coverage of the application through UT (Will cover more about Code Coverage in a little while).
Functional Testing
Functional testing is the major level of testing that's been concentrated by all the QA folks across the org. The method or the way in which testing is carried out has been covered in Testing Methods i.e. in the first few paragraphs of this blog.
Integration Testing
Integration testing can be viewed in two aspects the first one deals with the upstream systems and the second one deal with the downstream systems. Most of the time upstream system integration's issues or bugs are uncovered in functional testing. Whereas, in the case of downstream systems we might not be aware of the consequence of updating the existing flows. There is a common view among most of the Quality Assurance folks that it's the responsibility of downstream system team to take care of integration issues. Ideally, it's the responsibility of upstream system to make sure that there is no issues while upgrading the flows or we should inform them regarding expected outage or breakage when there is a change.
Test Coverage
Test coverage is one of the metric which has been perceived by QA folks as an area that needs to be covered by developers with unit tests. In reality the metric has to be maintained by QA team that includes dynamic coverage as well not much are really aware of how get the code coverage in dynamically and this is the area that needs a strong awareness.
Testing Tools
There is new testing tool that might be built or released by the time you are reading this blog but, how many ever tools that's been rolled out we should do a legitimate study or analysis before we pick up them in the stack of tools or technologies that's been in use already.
Security Testing
Not much of QA folks are aware of tools that help with testing security loop holes or the kind of security issues that might pop-up in the projects. I firmly believe that every one in the organization should have a certain knowledge on security testing practices.
Load Testing / Stress Testing
A Testing area which is also seen as a specialist job. And there are few so called specialist have presumption that the load generated is directly proportional to the number users that's been configured or used in the settings.
Requirement Traceability Matrix
Test Schedule
Test Records
Test Summary
Risk & Mitigations
All above areas are getting endangered as we have moved to so called agile life cycle.
