Performance Engineering

Performance engineering within systems engineering, encompasses the set of roles, skills,  activities, practices, tools, and deliverable applied at every phase of the systems development life cycle which ensures that a solution will be designed, implemented, and operationally supported to meet the non-functional performance requirements defined for the solution. As the connection between application success and business success continues to gain recognition, particularly in the mobile space, application performance engineering has taken on a preventative and perfective role within the software development life cycle. As such, the term is typically used to describe the processes, people and technologies required to effectively test non-functional requirements, ensure adherence to service levels and optimize application performance prior to deployment.Adherence to the non-functional requirements is also validated post-deployment by monitoring the production system.
Objectives

• Increase business revenue by ensuring the system can process transactions within the requisite timeframe
• Eliminate system failure requiring scrapping and writing off the system development effort due to performance objective failure
• Eliminate late system deployment due to performance issues
• Eliminate avoidable system rework due to performance issues
• Eliminate avoidable system tuning efforts
• Avoid additional and unnecessary hardware acquisition costs
• Reduce increased software maintenance costs due to performance problems in production
• Reduce increased software maintenance costs due to software impacted by ad hoc performance fixes
• Reduce additional operational overhead for handling system issues due to performance problems.

Approach

Evolving Customer Performance Requirement

Mostly the customer performance requirement are unambiguous and unrealistic. This is more clear from my recent experience with one of our customer. The key challenges are:-

1. performance requirement are very high level with accepted Metrics / workload not defined.
2. The tools and procedure for Performance evaluation is not defined.

Lets take each of this and see how we can involve customer in continuous engagement and avoid the last minute rush!

The requirement are defined with desired number of users and the acceptable response time. The initial gap we found around the distribution of users and the workload associated. Are these the realistic work load or are stressing the system in wrong way! Are we confirming to Customer needs which we cannot achieve! We initially told the customer all these issues and though we did not got complete details but we got two things the workload scenario and distribution. On seeing the scenario we immediately knew these are not realistic scenario and achieving the response time criteria with this scenario is not possible based on our experience but we went ahead so we could gather enough information on additional data like Hits/sec and Throughput and do compare against the market standard for similar tools.After multiple round of test we did various performance tuning but could not achieve the desire expectation. But we had enough data to show to customer that their needs are not realistic.

The was next challenge on the procedure involved for evaluating. The think times b/w transaction was 7-10 sec , user ramp up was not clear , peak load time not defined , content checks were done at each step , run time settings were completely default and all these were making the test failed at much lesser concurrency. After some discussion one of key difference was content checks and that was adding to the Load Runner response time as statement was executed for each vuser.

I suggest to add these learning to your case to better manage your customer requirement!

Evaluating Performacne the Customer Way!

We currently ran the performance testing for one of our prestigious customer.The tool we used for the test was load Runner . The test was supposed to be by two team one our team and another Customer team. We started the test on Customer database with 2500 CC and it got failed. The same test when run on our
internal Performance database and we have found that the test was scaling easily to 2500 CC and there comes out first learning

1. Evaluate the Volume of Database against when test is been done.
2. Evaluate the scenario against which UAT sign off is required.

Initial test runs failed and after multiple fixes we were able to successfully achieve 2500 CC. By the time we finished this test customer came up with his script and think time and we modified the script according to their think time and test again failed.Here comes my second learning

1. Never assume the think time even you are SME.
2. Hits/sec under certain concurrency is what Customer looks for.

Apart from the think time the customer did the URL mode scripting while we were doing the HTML based , currently we are running the test with context sensitive while out
Customer is running with Context Insensitive mode.

One can read moe about the Context Insensitive vs Context Sensitive on the blogs