Today, I wanted to share an experience that I had using Login Enterprise in a different way.
Some of you may know that I worked for Login VSI as a Services and Support Engineer. A few months after I left the company, a recruiter reached out to me. She was looking for candidate with experience with Login VSI/Enterprise for a client of hers, major banking company in the US.
This company had a large number of applications that they provided for their customers that spanned many markets outside of the finance realm, including healthcare, travel, etc.
As all good organizations do, they had requirements to perform regression testing for newly adopted applications as well as any upgrades. This testing entailed functional testing (making sure the functions and controls work as designed) and performance testing (making sure the functions do not exceed the expected times to respond.
They had recently adopted VMware Horizon as a delivery platform for Windows based applications, delivering them in a seamless window. During the transition to VMware Horizon, it was discovered that the tool they used to automate the performance testing would not work with this platform. This forced the application owners to perform these tests manually, having to recruit 100 or so employees to dedicate a day or week to following a scripted set of operations in their application and record the time it took to perform them. So, after doing a Proof of Concept with the Horizon Team, Login Enterprise was purchased to fill this niche.
Not your grandfather’s performance testing
Once I had accepted the position, it became evident that the methodology used for performance testing of the applications did not exactly align with how Login Enterprise performs tests. Below, I will discuss the requirement, why Login Enterprise did not fill that requirement and how we resolved the issue.
Of the three types of testing available in Login Enterprise, Load Testing was the obvious choice as we wanted to through a large number of users at the application.
Simultaneous testing
The first requirement was that all test accounts needed to run their scripts at the same time. This matches the behavior of users and what the application teams were doing during the manual performance testing.
Load Testing has you specify a ‘ramp up’ time to prevent overtaxing the infrastructure with excessive simultaneous logons, logging in ‘x’ number of users of ‘y’ amount of time. This was not suitable for our purposes. For instance, if we had a workload script that took 10 minutes to execute and we launched 500 users over 30 minutes, the first set of users would have completed their runs before the final users even logged on.
What we did was to provide a network share that was common to all of the test users. The workload script was designed to check a ‘flag’ file on that network share. Each test user that logged in would register its presence in the file. Then it would check to see if the required number of test users for that run had checked it. The script would continue to read that script every ‘z’ number of seconds to see when all test users had checked in. Once the required number of users had, then they would all proceed running through the script.
This required using a few extra C# libraries for the file/folder functions and worked well.
Different data for each user
This one might be fairly common to other organizaions…
Many of the applications that we were testing required the test users to actually modify records in the application (all testing was done in the Pre-Prod environment). Therefore, multiple users could not be working on the same records at the same time. At the time Login Enterprise only provided the ability to pass 2 distinct values to the script. Even though that has since been expanded, Managing that data at the Account object level was sub-optimal.
Instead, we decided to expand the concept that we used above. We created a CSV file with all of the data that the workload script might need and assigned each record to the test users as they logged in.
Different workflows in a single run
Another requirement of the testing was to have a portion of the test accounts doing one workflow while another set performed a separate workflow.
For instance, if the application was a travel reservations application, 120 of the 300 test users might be simulating the role of a reservation agent, looking up a customers details, checking flights and costs, etc. Another group of test users might be emulating a Customer Service rep, looking up current itineraries, etc. And another group of test users might emulate the Accounts Receivable, processing billing and issuing invoices.
Load Tests do allow you to specify multiple workloads, but all users would run all workloads and they would run synchronously.
To get around this, we would write each workflow as a separate script using Application X-Ray and the Script Editor tool. Once those were all tested (and debugged!). we would combine them into a single workload script. Using the above CSV file, we would dictate which users would run which workload and then add the logic to the workload script to define which routines in the script each users would run.
This actually worked really well, but it did really require working closely with the applications teams to define the requirements in great detail.
Reporting
The last thing I’ll talk about was the reporting of the test results. All of the reporting built into Login Enterprise for Load Testing to is to reflect the performance at a given time point in time. So, when you look at the charts, you can see the application timer value for that point in time.
What we needed was to report on what the Max, Average and 90 percentile for each timer. And even though the users were processing the script at the same time, specific operations would not be simultaneous.
Thankfully, we had the option to download the raw data as a CSV file. We used a script to process the data and for each timer, calculate and report the specific values.
Summary
So, with a little bit of creative thinking, a deep understanding on how Login Enterprise works and detailed requirements, you can use Login Enterprise for situations that may not match the original intended purpose.
Leave a comment if this helps you think of other use cases for Login Enterprise or if you have already used it for an ‘Out of the box’ solution.