Telemetry data, which are units of data (hardwired/gameplay metrics) embedded into the game via code that are then transmitted to a collection server. This data can help answer questions about player behavior and how players are experiencing games on a level that can be difficult or impossible to obtain through direct observation. Example behaviors caught by telemetry in an action game, for example, can include units such as player deaths, number of enemies killed, abilities used, weapon/ammo pick ups, or mission completion time (as in the following example). One primary use of telemetry data, which can be collected during a playtest or during another period of play (e.g., closed/open alpha or beta, as well as post-launch), is for post-playtest analysis. In the following example, a developer was curious how long it took players to finish the missions they played. This dev had an ideal length in mind for each mission and this data would inform next steps to help reach that vision.
The figure shows the data split by each player (x axis, n = 8) and their completion time per mission (y axis, in minutes).
Following more iteration and playtesting, additional players can be added to these analyses to increase the sample size and provide an average completion time per mission. Averages are not super useful with small sample sizes (here n = 8) since outliers can easily pull the average in a certain direction, skewing the results. And you can see from the graph that the range of time it took playtesters is quite varied within missions.
Furthermore, data can be collected during longer periods of play, such as an alpha or beta period, with averages calculated throughout as well how the average mission completion time may have fluctuated from one play period to subsequent ones (e.g., pre-alpha playtests –> closed alpha –> closed beta –> open beta). This will give devs a better representation of the execution of the mission design and how iteration has helped manipulate its length on a larger scale.
This is one example of how telemetry data collection during playtests can be useful. Specific queries can be generated to pull the relevant data from the server and then visualized for consumption by the dev team following the playtest. In turn, this data can help inform and drive next steps for designing specific game elements to ensure players’ experience is as close to the intention as possible.