Mostly the customer performance requirement are unambiguous and unrealistic. This is more clear from my recent experience with one of our customer. The key challenges are:-
1. performance requirement are very high level with accepted Metrics / workload not defined.
2. The tools and procedure for Performance evaluation is not defined.
Lets take each of this and see how we can involve customer in continuous engagement and avoid the last minute rush!
The requirement are defined with desired number of users and the acceptable response time. The initial gap we found around the distribution of users and the workload associated. Are these the realistic work load or are stressing the system in wrong way! Are we confirming to Customer needs which we cannot achieve! We initially told the customer all these issues and though we did not got complete details but we got two things the workload scenario and distribution. On seeing the scenario we immediately knew these are not realistic scenario and achieving the response time criteria with this scenario is not possible based on our experience but we went ahead so we could gather enough information on additional data like Hits/sec and Throughput and do compare against the market standard for similar tools.After multiple round of test we did various performance tuning but could not achieve the desire expectation. But we had enough data to show to customer that their needs are not realistic.
The was next challenge on the procedure involved for evaluating. The think times b/w transaction was 7-10 sec , user ramp up was not clear , peak load time not defined , content checks were done at each step , run time settings were completely default and all these were making the test failed at much lesser concurrency. After some discussion one of key difference was content checks and that was adding to the Load Runner response time as statement was executed for each vuser.
I suggest to add these learning to your case to better manage your customer requirement!
1. performance requirement are very high level with accepted Metrics / workload not defined.
2. The tools and procedure for Performance evaluation is not defined.
Lets take each of this and see how we can involve customer in continuous engagement and avoid the last minute rush!
The requirement are defined with desired number of users and the acceptable response time. The initial gap we found around the distribution of users and the workload associated. Are these the realistic work load or are stressing the system in wrong way! Are we confirming to Customer needs which we cannot achieve! We initially told the customer all these issues and though we did not got complete details but we got two things the workload scenario and distribution. On seeing the scenario we immediately knew these are not realistic scenario and achieving the response time criteria with this scenario is not possible based on our experience but we went ahead so we could gather enough information on additional data like Hits/sec and Throughput and do compare against the market standard for similar tools.After multiple round of test we did various performance tuning but could not achieve the desire expectation. But we had enough data to show to customer that their needs are not realistic.
The was next challenge on the procedure involved for evaluating. The think times b/w transaction was 7-10 sec , user ramp up was not clear , peak load time not defined , content checks were done at each step , run time settings were completely default and all these were making the test failed at much lesser concurrency. After some discussion one of key difference was content checks and that was adding to the Load Runner response time as statement was executed for each vuser.
I suggest to add these learning to your case to better manage your customer requirement!
No comments:
Post a Comment