Improving Product Discovery of Tabletop RPG Maps (Preliminary Investigation)

2025-06-04

As catalogs for digital marketplaces grow in size, customers have greater difficulty in finding products that meet their needs. Marketplace owners might improve product discovery by adding new categories, tags, or other searchable and filterable metadata to product descriptions. As it is often difficult to get product owners to update old product descriptions, a marketplace owner may use automated methods to “backfill” product data. We investigate using multi-modal computer models to extract data from digital products, specifically, digital maps for tabletop role playing games. Across five tasks, we find that models perform comparatively with their ranking on the Open VLM Leaderboard. State-of-the-commercial art models perform well with zero-shot feature extraction and image association/captioning tasks, but all models perform poorly with reasoning and quantitative tasks.

Introduction

Tabletop role playing game maps primarily serve two roles: a way for players and the game master to track positions of characters and creatures via miniatures or tokens (see battlemap below), or a depiction of an area for the game master’s use as they run an adventure which denotes buildings, catacombs, traps, and locations of creatures (see dungeon map below). The former is either printed and placed in the center of the table during play or displayed using virtual tabletop (VTT) software.

Game masters need to marry a map with their intended adventure and campaign setting, so the choice of a map is multi-factor decision. Although game masters often make their own maps for an adventure, high-quality maps require skill, creativity, and can be time-consuming to make, so game masters may instead purchase maps. Multiple marketplaces exist to sell maps created by independent publishers. The marketplaces support product discovery via categorization filters, free-form text search, and limited recommendation systems.

Since maps are digital goods and marketplaces have access to the product (marketplaces store the product within their databases), a marketplace may use computer vision technology to extract product data and augment their product discovery tools and database. As a means to maintain accuracy, publishers could approve any additional data being added to the product description or database. A possible activity flow for this process could be:

1. Publisher creates product description and uploads product to Marketplace
2. Model (mediated by the Marketplace) processes the product and computes additional metadata
3. Publisher examines and approves additional metadata
4. Customers use tools provided by the Marketplace to discover products

Computer vision technology has improved greatly in the past few years, particularly in its ability to identify and extract objects and other features from an image. The combination of disparate neural network architectures allows model to interpret image data with semantically-laden text, greatly improving performance on tasks such as image captioning and visual question and answering.

Methodology

As a preliminary study, our objective is to gain a sense of the current quality of existing models for this domain, gain insight on what engineering efforts will be required for a large scale project, and characterize some of the likely challenges such a project would encounter.

We evaluated zero-shot performance of five existing multi-modal neural networks using five prompts or tasks. The prompts were:

1. List locations or spaces featured within the map. Format as a bullet list.
2. List items featured within the map. Format as a bullet list.
3. In units of the listed map scale or map grids, if they exist, what are the dimensions of this map?
4. Is movement within this map mostly linear or mostly branching?
5. List types of adventures or scenarios this map is suitable for. Format as a bullet list.

The first two prompts focus on identifying and extracting features, the former focusing on locations where characters may be placed, while the latter on items the characters might pick up or interact with. The third prompt involves interpreting, quantitatively, a map grid or scale. The fourth prompt involves semi-subjective interpretation of the map features along with the external concept of movement. The fifth prompt tests themed open-ended captioning.

We selected five multi-modal LLMs for the evaluation (see table below). Since we would like to determine a necessary “minimum strength,” we selected models across a variety of benchmark levels. The Avg Score column in the table acts as a proxy for the model’s general strength (source: OpenVLM Leaderboard).

All models were evaluated using the same core prompts. We adjusted the prompts to fit model requirements (e.g. adding ‘USER’ and ‘ASSISTANT’ breaks), but the core prompt was constant.

We did not supply contextual data to any of the models. Each prompt was presented as a stand-alone invocation rather than a conversation to avoid interacting effects. Some models required converting images to RGB mode first, but we otherwise did not scale, crop, or convert the images for a specific model.

For our dataset, we randomly selected products from a digital marketplace. All products were created between 2022 and 2024, inclusive. We dropped products from our selection if they were city or world maps or were not maps (e.g. collections of images meant to build new maps). Since most products include multiple maps and map variants (e.g. day and night), we selected a single map from each product, preferring ones meant for VTT use and variants with labeled content.

We ran the three open-source models on a 16-core, 64-gb virtual machine without an attached GPU. Our code used the HuggingFace transformer library. We called the two API models using their commercial endpoints.

Results and Evaluation

For each of the five prompts, we evaluated the quality of output with prompt-specific measures and human judgement. Although we selected the sample images randomly, this study does not include sufficient data points for a signficance test. Instead, as a preliminary study, we want a rough measure of baseline capability in order to identify potential research directions and likely necessary data engineering efforts.

For transparency, we report the output of the models for each task within a table. However, the output has been edited for uniformity and space. The models often output filler sentences, repeat themselves, and change formatting. Any large-scale application of these models will require an engineering effort to standardize the outputs.

Inference Time and Money

Inference times in seconds, not including any file upload times, were:

Our intent here is not to rigourously benchmark inference performance but to give a sense of the scale of latencies. The APIs, which are presumably tuned and run on high-end machines, needed thirty to sixty seconds to evaluate five prompts on a image. Run locally on a high-end machine that lacks a GPU, inferences took multiple minutes to complete.

The marginal cost of calling the API services was between $0.002 to $0.003 per image. The APIs use a variety of factors to calculate a cost in terms of tokens, but this calculation is dominated by the size of the image in pixels.

To compute the marginal cost using for local inference, we use the cost of running a virtual machine per hour and processing throughput. The virtual machine used for this study costs $0.54/hour. This translates to costs from $0.018 to $0.18 per image processed, depending on the model.

We expect an operator could process 10,000 images for less than $100.

Prompt 1: Locations

List locations or spaces featured within the map. Format as a bullet list.

This task tests the model’s performance at finding map features that are conceptually a “room” or defined space. Characters should be able to occupy a location. For each response, we apply a subjective criteria that the list describes rooms that are accurate and meaningful in a product query.

We’ve summarized the results into the table below. Both Gemini and GPT extracted useful details consistently from the images, although tuning or other filters may be required to make the outputs useful. The three hosted models typically had poor quality. When labels were present, all models preferred to extract those labels rather than interpret details in the image.

Details

M001 shows the deck of a ship. There is one room to the stern with a hammock, table, barrles and a crate. There is also a small room to the stern holding an anchor. The middle area hosts four cannon, two to a side, along with more barrels, crates, a ladder, and a grate. The aft is separated into two rooms, each with a hammock and a table with chair. All the models pick up on the naval theme. Pixtral picks up on the contents of rooms, but none of the rooms themselves. Gemma3 reports a “Captain’s Quarters” and a “Navigation Station,” but it is unclear what room is affiliated with either description. Similarly, Gemini uses general terms to refer to sections of a boat, without reference to specific rooms on the map, and reports “lifeboats/rafts”. GPT fares better by referring to “Crew quarters” and “Private cabins” without suggesting specific roles.

M002 is a cliff-face, dominated by a large skull where the mouth opens into a cave. The cliff face is embedded with single-room buildings connected by stairs. The ground features a small camp and some streams crossed with bridges. None of the models pick up on the skull imagery, although all models except Pixtral report an entrance to a cave. Gemma3 fails to identify the buildings and the camp while LLaVA combines these features into a generic “small village” label as well as scales the small rocks in the streams into islands. GPT provides the most complete and detailed list, followed by Gemini.

M003 is a large tree with a small opening that leads to an interior, hollowed-out room that is likely a lair for a creature. LLaVA directly extracts the hollowed-out nature of the room (albeit encounters a model loop and keeps repeating the fact). Gemini reports the fact somewhat indirectly, referring to it as a “central clearing/nest”. The other three models report the existence of the tree and note other elements of the image, but not the interesting part of the image — the hallow.

M004 is a complex map, featuring a bridge leading to a fortress of rooms, with internal paths and stairways connecting detailed rooms. LLaVA fails and simply repeats “A dungeon with a staircase” repeatedly. Other than Gemini, the other models associate rooms with functions in a non-apparent manner. Is a room with a table and three seating places the “dining hall”? Which room is a “jail” or “prison”?

M005 is a difficult map for a human to interpret since the color range is low contrast and the buildings are largely collapsed with few distinguishing marks. Of the open source models, Pixtral does best by listing “structures”. None of the open source models identify the stream or river in the picture, but both API models do. Gemini’s descriptions are matter-of-fact, while GPT’s are more detailed and add color (not necessarily supported by the image).

Both M006 and M007 use traditional RPG cartography as found in TSR publications in the 80s and 90s (similar to the GM map above). Rooms are labeled and, while doors, traps, and secret passages are marked, rooms are not populated with images of their contents. Gemini and GPT extracted all the room labels and other markers from the map, however, GPT also added labels to each room number with descriptions that fail to follow the actual room layouts. Of the three opensource models, Pixtral performed the best by extracting 80% in M006’s case and one-third of the rooms in M007’s case. From a product discovery, extracting room labels is not useful by itself, as customers are not searching for a map that includes “room 33”. However, customers may be interested in maps with a certain number or range of rooms, so the output may be useful as part of an analysis pipeline.

Map M008 is a octagular bar or tavern. Gemma3 and Pixtral extract elements of the image, but fail to extract the meaning of the elements. LLaVA, Gemini, and GPT identify the bar and the other dining/drinking elements.

Responses

Prompt 2: Items

List items featured within the map. Format as a bullet list.

This task is intended to evaluate sub-image identification within a larger image, focusing on images that are “items” — things characters could interact with, versus rooms the characters could inhabit. This is a feature associated with “battle maps” or maps that will be seen by players, versus maps only seen by the game master. Thus, we remove M005, M006, and M007 from the evaluation of this task. M005 is excluded because the maps lacks details of items and M006 and M007 are not battle maps.

This task led the LLaVA model to repeat itself. For clarity, we have removed duplicate entries.

For this task, we use Jaccard similarity, also called intersection over union, to quantify how effectively the models are finding all the items. The score ranges from zero to one, where one would indicate the model replied with all of the items in the ground truth and without any extras. Mathematically:

$$ J(A, B) = \frac{|A \cap B|}{|A \cup B|} $$

For our evaluation, we ignore plurality and counts. For example, “cannon” is treated as a correct answer for M001, as well as “cannons” and “cannon (6)”, even though there are only four.

The scores for the models on M001, M002, M003, M004, and M008:

For the open source models, their errors were often ones of omission – not reporting elements of an image. In contrast, the API model scores were damaged by reporting non-existent elements or reporting elements that did not fit within the definition of an item. This is a case where further prompt engineering might significantly improve the quality of the answers.

Responses

Prompt 3: Grid Dimensions

In units of the listed map scale or map grids, if they exist, what are the dimensions of this map?

This task involves identifying a map feature (a grid) and then counting squares. If a model performs well at this task, a marketplace might be able to offer geographical information system-like filters.

All of the models failed this task. The two most advanced models, Gemini and GPT, successfully gave a “cannot be determined” answer when the maps lacked a grid or any scale marker. (Contrast this with “non answer” which uses filler phrases.) However, they were otherwise highly unreliable with their grid counts, even if they were close (numerical difference) in some cases.

This task will be difficult for LLMs. First, since LLMs work on a tiling of the source images, the image will usually be distorted in order to be standardized. Second, this task involves counting which neural networks are not ideal for. If this was an important element for product discovery, we expect a specialized model using a convolutional neural network (CNN) to extract the grids, or perhaps a more direct technique (e.g. Hough transform), would be efficient and reliable.

Prompt 4: Linear or Branching Movement

Is movement within this map mostly linear or mostly branching?

This tasks involves a higher-order understanding of the graph or connected nature of rooms, along with an understanding of how characters move along the network. Practically, this distinction is an important element of adventure design.

If we treat this task as a boolean classifier, Pixtral and Gemini both scored the best with 5 correct out of 8. GPT scored next with 4 correct out of 8 and Gemma3 and LLaVA scored 3 out of 8. (We graded Gemma3’s non answers as incorrect answers, but they could be graded more severely.) Overall, the results suggest performance no better than a coin flip. However, this is a subjective task and we expect that if more humans were involved to make a labeled dataset inter-rater reliability would be low.

The reasoning text provided by Gemini and GPT suggests the models are recognizing the graph-like nature of the maps and how movement flows through passages and doors. With further experimentation and a more developed dataset, higher performance may be unlocked.

Prompt 5: Scenarios

List types of adventures or scenarios this map is suitable for. Format as a bullet list.

This task is an open-ended generation of adventure ideas or classifications. This is similar to image caption generation. Alternatively, a marketplace owner could turn this into a classification problem by restricting the list of scenarios to a defined set. Publishers, of course, could use this technology to augment their own product descriptions with lists of creative ways the maps could be used in games.

As an open-ended generation task, we will use a metric modeled after signal-to-noise. (A method like CIDEr would be more rigorous, but more expensive to implement.) Each adventure or scenario line will be assigned to either the “signal” or “noise” category. The score will be the ratio between the count of the “signal” responses over the count of the “noise” responses plus one (to avoid division by zero). Mathematically:

$$ h = \frac{|s|}{|n| + 1} $$

Gemini and GPT both excel at this task. Gemini’s responses tend to be more “classification”-oriented while GPT tends to be more of a story prompt, although neither are fully consistent. Gemma3 scores low because the model outputs few responses. Pixtral has more consistently higher signal than LLaVA.

Conclusion

Multi-modal large language models exhibit a range of quality. Product discovery applications will likely require, at least, models with late 2024 or 2025 state-of-the-art performance. For VTT-style maps, models can yield lists of rooms and items found within a map, although individual entries may be irrelevant for purpose and the lists often contain hallucinated or misinterpreted items. For GM-style maps, models extract labels with high-recall and precision, but other details may be imprecise. Our tests showed all models were poor at understanding scale or grid systems, although some could detect the existence of a grid. Similarly, reasoning about movement through the map may be beyond current LLMs, although we suspect this area needs more rigorous study. LLMs scored highly on generating scenario or story ideas tied to a map, suggesting this might be a straight-forward product discovery mechanism.

Notes

The dataset was not incorporated as training material into any model.