Ntrance

Supporting global players

Testing the performance of online multiplayer games is notoriously difficult. Player numbers are hard to predict, and users often interact with the game and with each other in ways that game designers never anticipated. When it comes to a complex massively multiplayer online role-playing game (MMORPG) like DK Mobile, the challenge is compounded by the large in-game world, the numerous items and customization options, and the open gameplay model that gives many thousands of simultaneous players freedom to choose their own paths through the game.

Kim Byung Su, CEO of Ntrance, says: “We support up to 100,000 players simultaneously from a global userbase spanning multiple time zones—so DK Mobile must operate responsively and reliably, 24/7. While the game is free to download and play, we generate revenue by enabling users to buy and sell in-game items, so performance issues or downtime have a direct financial impact for our business.”

Aiming to maximize game server performance and reliability, Ntrance wanted to test a number of likely in-game scenarios—in particular, item purchases and gacha (a mechanism whereby players spend in-game money and receive a random in-game item in return). These activities are key sources of revenue for the company, so it's vital to keep them running smoothly and consistently.

“The nature of the game means that we can't do performance tests based on transactions; we have to conduct performance tests based on simulated user actions,” says Kim Byung Su. “Our game was developed using Unity and C# on .NET, so we needed a performance testing platform that fully supported .NET protocols.”

It was almost impossible to accurately test performance with other solutions, but LoadRunner offered complete support for .NET.

Simulating complex game scenarios

Working with Cleversoft as its testing partner, Ntrance chose OpenText™ LoadRunner™ Professional as its strategic platform for performance testing. “We had found that it was almost impossible to accurately test performance with other solutions, but LoadRunner offered complete support for .NET,” says Han Min Ho, Lead Developer at Ntrance. “LoadRunner also supports the shift-left philosophy by making it easier for developers to get involved in performance testing. It provides an intuitive interface that resembles an IDE, and its load control and highly accessible interface for analyzing results are helping us to reduce testing time.”

Ntrance deployed the LoadRunner Controller plus ten Load Generators on the Amazon Web Services (AWS) cloud; meanwhile, the target environment for performance testing was running in a separate AWS environment. The target environment includes a game engine (IRUN Factor, a game server used widely in South Korea), a user authentication server, and a database server.

“The asynchronous style of progression in the game creates a heavy testing workload,” says Han Min Ho. “We needed up to 25 Load Generators, each on an 8-core AWS instance with 32 GB of memory, to simulate the load of up to 5,000 concurrent virtual users—which is the number of users we typically support on each production instance of the game engine.”

The complex nature of the game also made it impractical to use the native recorder in LoadRunner Professional to create testing scenarios. Instead, Cleversoft and Ntrance used the LoadRunner Professional .NET framework to create tests programmatically.

“From the point of view of our .NET developers, it was simple to write tests for LoadRunner,” says Han Min Ho. “We create virtual users with VuGen scripts written in C#, and LoadRunner connects them to the game server to perform scripted sequences of in-game actions.”

Ntrance uses LoadRunner Professional to run several different performance testing scenarios, from ten-minute unit testing of function calls to the game server all the way through to eight-hour soak testing of long-running complex scenarios to check for system resource leaks over time.

Han Min Ho says, “LoadRunner enables us to simulate any set of real-life actions that a player might perform. We selected the 28 most commonly used actions for performance testing and created detailed scripts for each. Those scripts are stored and can then easily be re-used.”

As an example, one of the 28 scripts is “Game Connect”, in which the virtual user queues to log in, connects to the game server, creates a character and then accesses that character. A more complex example is the “Guild” script (a guild is a voluntary grouping of players with common objectives), which covers: guild creation, guild attendance, guild donations, registration/deletion of hostile guilds, appointment of arbitrary user corps commanders, placement/release of voluntary corps characters, instantaneous movement to guild members, move to arbitrary guild sharing positions, withdrawal, and disbandment.

When the performance testing scripts are running, Ntrance monitors resource usage including virtual CPU and memory on AWS. Testing has shown that the performance of DK Mobile scales in a very linear way: CPU usage with 5,000 users on one game engine is 50% on a 32-core AWS instance versus 95% on a 16- core instance. Performance testing has also highlighted delays in character loading time, and instabilities caused by limited OS network buffer capacity when too many users teleport to the same location at the same time. Based on these findings, Ntrance developers have been able to optimize performance by adjusting the game logic.

Naturally, the number of frames per second delivered to the user's screen is a critical measure in online gaming. A low frames per second count creates a jerky and unconvincing experience, reducing player immersion in the virtual world and making them less likely to enjoy—and spend money in—the game. Ntrance monitors frames per second using mobile emulators such as Nox and BlueStacks.

In addition to providing live dashboards so that developers and testers can monitor performance during test execution, LoadRunner Professional captures and stores huge volumes of data for post-test review in LoadRunner Analysis. Metrics are collected and displayed in separate graphs during runtime execution, and LoadRunner Analysis then allows users to merge and overlay graphs and perform deep-dive investigations.

“In LoadRunner Analysis, we primarily look at graphs of response time, concurrent users, and resource utilization—and the Merge function has been very helpful in comparing graphs,” says Han Min Ho.

Gaining visibility, saving time

Using LoadRunner Professional to test the performance of DK Mobile has enabled Ntrance to optimize the performance and stability of its software, ultimately helping to enhance player loyalty and increase revenue from in-game purchases.

Despite the complexity of typical gameplay scenarios in DK Mobile, LoadRunner Professional is able to provide full coverage of the key factors that Ntrance wants to test, including the consistency and integrity of the in-game economy.

“Consistency testing helps us to fix bugs, avoid crashes and eliminate bottlenecks in the user experience,” says Han Min Ho. “We can see some results right away, then perform deeper analysis of graph trends to better understand irregularities.”

Performance testing is increasingly a team sport: with the shift-left movement, developers are getting involved earlier in the process. “For our performance engineers, the Micro Focus (now OpenText) solution solved our scripting challenges, making it easier to find and resolve performance bottlenecks,” says Kim Byung Su. “And for our developers, the recommendations from LoadRunner help them focus in on the right place to fix the issues.”

LoadRunner Professional provides useful real time information on all transactions performed by virtual users in the game. Thanks to the live dashboard view while performance tests are running, Ntrance can collect and view metrics related to key game scenarios and see unexpected behavior whenever it occurs. Kim Byung Su says: “Real-time data has been very helpful in understanding the current game progress, such as the number of item purchases and the number of monster kills in battle.”

In general terms, test scripting and the analysis of results are the two most time-consuming practices in the life of a performance tester. LoadRunner Professional accelerates both, saving considerable time and effort for Ntrance. “In my experience, LoadRunner compares very favorably with open-source tools such as JMeter: I estimate that we've reduced the total testing time by one month or more,” says Han Min Ho.

Kim Byung Su concludes: “I believe that LoadRunner should be recognized as the best solution in game performance testing. It supports a variety of protocols that other solutions have not implemented, and industry studies show that it is well-positioned as an industry leader. We're confident that the reliability of the results we get from LoadRunner is second to none.”

LoadRunner should be recognized as the best solution in game performance testing… We're confident that the reliability of the results we get from LoadRunner is second to none.