https://revisit.dev/PR232/blog/Home | ReVISit Blog2026-04-30T00:00:00.000Zhttps://github.com/jpmonette/feedHome | ReVISit Bloghttps://revisit.dev/PR232/img/logos/favicon.svg<![CDATA[Integrating an LLM-Based Chatbot into a reVISit Study]]>https://revisit.dev/PR232/blog/2026/04/30/llm-in-revisit/2026-04-30T00:00:00.000ZStudying how people interact with AI is increasingly relevant, and when that AI is embedded in an interactive system, such as a data visualization, the questions get even more interesting. With reVISit, you can now integrate an LLM-based chatbot directly into your study workflow. In this post, we walk through a working demo and show you how to adapt it for your own research.
Take a look at our demo study. The goal of this scenario was to help people learn about a chart. In this context, the chatbot should do more than simply generate answers. It should understand the study context, respond based on relevant materials, maintain continuity across turns, and make every interaction traceable for later analysis.
Below, we describe why this chatbot is useful for studies, highlight its core features, and explain how to customize it.
In the demo study, participants can ask questions about a clustered heatmap, and the chatbot responds with streaming text. When a question requires exact data values or visual details, the chatbot can request the dataset or chart image. All inputs and outputs are tracked by Trrack, our provenance library, making it possible to analyze how participants interact with the chatbot and what information they request.
Our example uses the OpenAI Responses API and is implemented with the following features:
Conversation memory: keeps the model aware of prior turns
Streaming responses: reduce perceived latency and make the interface feel more responsive
Sending the chart image and dataset only when needed: gives the chatbot access to study materials only when required
Recording provenance and conversation history: makes interactions traceable for later analysis
The demo is designed to be flexible (check out the code). Most settings live in ChatInterface.tsx, where you can customize the chatbot’s behavior and study materials.
One challenge with LLM-based interfaces is keeping API keys secure in a (mostly) serverless environment such as a reVISit study. Our demo uses a proxy server to get around this problem. We created a simple proxy server for you, which you can find at https://github.com/visdesignlab/openai-api-proxy.
After deploying the proxy, set VITE_OPENAI_API_URL in your .env file for either local development or production.
If you have questions or want to share what you build, come find us in the reVISit Slack. We’d be excited to see how you apply these ideas in your own prototypes.
]]>Tingying Hehttps://tingying-he.github.io/Alexander Lexhttps://alexander-lex.net/<![CDATA[ReVISit 2.4: All About Speed and Stability!]]>https://revisit.dev/PR232/blog/2026/02/23/release-2.4/2026-02-23T00:00:00.000ZVersion 2.4 of reVISit is out – and for once it doesn’t pack a lot of new features. Rather, it’s all about quality of life: we fixed bugs, improved stability, and, most importantly, made the documentation much much better!
We also added a few new things here and there, but they’re just quality of life improvements for supporting studies currently under development by the community.
We recommend that all reVISit users update to version 2.4, since it should provide a better experience to both developers and participants. However, if you have a study implemented in 2.3 that you’ve tested and piloted already and want to run tomorrow, you should probably stick with 2.3. After updating, make sure to run pilots with 2.4 to see whether everything works as expected.
So, let's dive into the details of what’s new in 2.4!
As always, the recorded data will contain the condition information allowing you to easily differentiate between cases in analysis. You can also see all of the defined conditions and jump right into them from the study card, as shown above.
The key additions to get this functionality are the conditional flag and the id key. Make sure to make your ids unique!
With this feature, you’ll be able to use dedicated links to recruit participants for each condition (for example, using two different Prolific studies), and keep recruiting until you have reached the desired number of participants for each.
Sometimes you want to include the same component multiple times in your study. For example, you might want to measure task load after each task. This wasn’t possible in reVISit yet (you could hack it by using inheritance). Now, however, that’s easy to do: just include the same component multiple times. For example, if you include a component nasa-tlx after each task, you’ll have access to nasa-tlx-1, nasa-tlx-2, etc, in the data.
We previously supported referring to components of libraries as $libraryName.co.componentName as well as $libraryName.components.componentName, and similarly used .se or .sequences for sequence. The shorthand is now deprecated; new studies should only use .components and .sequences.
We have virtualized our study browser so that it does not render information for every possible study component at once. For studies which had large amounts of components (think 1000+) this was causing slowdowns during development.
We have also fixed a number of small bugs that impacted performance, especially for studies which contained large amounts of provenance data. Such studies should see a significant participant facing performance improvement in 2.4.
A key focus of this update has been to improve the robustness of our infrastructure. Previously we had focused on using integration level tests using the playwright test framework. With this update we added more than 300 unit tests, which improves the quality of our tests, makes the run faster, and more specifically.
With this release, we closed over 80 issues on GitHub. For details on the bugs that were fixed, take a look at the release logs.
]]>The ReVISit Team<![CDATA[Remote Usability Testing of Figma Prototypes with reVISit]]>https://revisit.dev/PR232/blog/2026/02/21/figma-prototype-testing/2026-02-21T00:00:00.000ZScheduling one-on-one usability sessions is time-consuming, expensive, and hard to scale. With reVISit's screen recording and think-aloud audio features, you can run instrumented prototype studies asynchronously — participants navigate your Figma prototype while their screen and voice are captured, and you can run multiple participants at once!
Usability testing can be difficult. You have to track people down, schedule a time for the session, and repeat for each user. Remote testing, e.g., via Zoom, is possible, but still requires you to be online at the same time.
ReVISit now offers an alternative through its new screen recording and audio think-aloud features. Because Figma provides shareable embed URLs for interactive prototypes, you can drop one directly into a reVISit website component using iframes — with very little coding needed. Layering on "recordScreen": true and "recordAudio": true gives you:
Screen recordings of every participant's navigation path through the prototype
Synchronized think-aloud audio so you can hear their reasoning alongside what they see
Structured questionnaires before and after the prototype task — in the same study, same dataset
Participant Replay to watch recordings back with audio aligned on a single timeline
The entire study is defined in a single JSON config, and all data lands in your reVISit storage automatically.
Step 1: Enable Screen Recording and Think-Aloud Audio
Start by turning on both capture modes in the top-level uiConfig block:
Then import the companion library that handles participant permissions:
"importedLibraries":["screen-recording"]
The screen-recording library provides a screenRecordingPermission component that asks participants to share their browser tab — similar to screen-sharing in a video call. When recordAudio: true is also set, the permissions step requests microphone access at the same time. ReVISit blocks progression if either permission is denied.
Insert the permission step near the start of your study sequence:
In Figma, open your prototype, click Share → Open embed code or Publish, and copy the URL from the src attribute of the generated <iframe> tag. It will look something like:
Paste that URL into a website component in your reVISit config:
"figmaTask":{ "type":"website", "recordAudio":true, "recordScreen":true, "path":"https://embed.figma.com/proto/<YOUR_FILE_ID>/<PROTOTYPE_NAME>?...", "instruction":"Please explore the prototype below. As you navigate, speak your thoughts aloud — tell us what you notice, what you expect, and anything that feels unclear or surprising.", "response":[] }
Setting recordAudio and recordScreen on the component itself is optional when they are already enabled globally in uiConfig, but it is good practice to be explicit — especially if you want different recording behavior on other components (e.g., a plain questionnaire where you turn audio off).
Step 3: Guide Participants with Sidebar Instructions
Without a moderator in the room, it can be easy for participants to forget to think aloud or to drift into their working process. It can be useful to include an on screen prompt to remind them to stay on task.
Add a textOnly response at "location": "sidebar" to keep instructions visible throughout the entire task:
"response":[ { "id":"think-aloud-instructions", "type":"textOnly", "location":"sidebar", "prompt":"### Think-Aloud Instructions\n\nYour screen and audio are being recorded. Please say out loud everything you are thinking as you move through the prototype.\n\n**As you explore, try to describe:**\n\n- What you notice on each screen\n- What you expect to happen when you tap or click something\n- What feels clear or confusing\n- Why you choose certain actions\n\nThere are no right or wrong actions — we are evaluating the design, not you." } ]
Sidebar content remains pinned while participants scroll or interact, so the instructions stay in view even as they move through multiple prototype screens.
Follow the prototype task with a questionnaire component. Setting "recordAudio": true on it means participants can continue to narrate their thoughts while they fill out rating scales and open-ended questions — a useful signal for interpreting their written responses. Alternately, you might design a series of pages with one spoken question per page, which will automatically be segmented by reVISit's trial management architecture.
"postTaskQuestions":{ "type":"questionnaire", "recordAudio":true, "response":[ { "id":"satisfaction", "prompt":"Rate your satisfaction with the following aspects of the prototype.", "location":"aboveStimulus", "type":"matrix-radio", "answerOptions":"satisfaction5", "questionOptions":[ "Overall Design", "Ease of Navigation", "Visual Clarity", "Completeness of Information" ] }, { "id":"open-feedback", "prompt":"What stood out most during your exploration — positive or negative?", "location":"aboveStimulus", "type":"longText", "placeholder":"Share your thoughts here..." } ] }
The Likert matrix response type (matrix-radio) with the built-in satisfaction5 answer options gives you a clean, pre-labeled five-point scale without any extra configuration.
Once participants complete the study, head to the reVISit analysis interface. The Participant Replay view lets you watch each participant's screen recording with their think-aloud audio synchronized on a timeline — so you can see exactly where they hesitated, backtracked, or expressed confusion.
For deeper qualitative analysis, the Qualitative Coding interface displays auto-transcribed think-aloud text for each participant and task, lets you correct transcriptions, and supports creating and applying codebooks to surface recurring themes across your dataset.
Here is a complete, ready-to-use config. The only change you need to make is swapping in your Figma embed URL for the path value in figmaTask:
{ "$schema":"https://raw.githubusercontent.com/revisit-studies/study/main/src/parser/StudyConfigSchema.json", "studyMetadata":{ "title":"Prototype Usability Study", "version":"pilot", "authors":["Your Name"], "date":"2026-02-21", "description":"A think-aloud usability study with screen and audio recording.", "organizations":["Your Institution"] }, "uiConfig":{ "contactEmail":"your@email.com", "withProgressBar":true, "withSidebar":true, "recordAudio":true, "recordScreen":true }, "importedLibraries":["screen-recording"], "components":{ "introduction":{ "type":"markdown", "path":"<PATH_TO_YOUR_INTRODUCTION.md>", "response":[] }, "figmaTask":{ "type":"website", "recordAudio":true, "recordScreen":true, "path":"https://embed.figma.com/proto/<YOUR_FILE_ID>/<PROTOTYPE_NAME>?node-id=...&scaling=min-zoom&embed-host=share", "instruction":"Please explore the prototype below. As you navigate, speak your thoughts aloud — tell us what you notice, what you expect, and anything that feels unclear or surprising.", "response":[ { "id":"think-aloud-instructions", "type":"textOnly", "location":"sidebar", "prompt":"### Think-Aloud Instructions\n\nYour screen and audio are being recorded. Please say out loud everything you are thinking as you move through the prototype.\n\n**As you explore, try to describe:**\n\n- What you notice on each screen\n- What you expect to happen when you tap or click something\n- What feels clear or confusing\n- Why you choose certain actions\n\nThere are no right or wrong actions — we are evaluating the design, not you." } ] }, "postTaskQuestions":{ "type":"questionnaire", "recordAudio":true, "response":[ { "id":"satisfaction", "prompt":"Rate your satisfaction with the following aspects of the prototype.", "location":"aboveStimulus", "type":"matrix-radio", "answerOptions":"satisfaction5", "questionOptions":[ "Overall Design", "Ease of Navigation", "Visual Clarity", "Completeness of Information" ] }, { "id":"open-feedback", "prompt":"What stood out most during your exploration — positive or negative?", "location":"aboveStimulus", "type":"longText", "placeholder":"Share your thoughts here..." } ] } }, "sequence":{ "order":"fixed", "components":[ "introduction", "$screen-recording.co.screenRecordingPermission", "figmaTask", "postTaskQuestions" ] } }
]]>Lane Harrisonhttps://web.cs.wpi.edu/~ltharrison/<![CDATA[ReVISit 2 Paper Wins Best Paper Award at IEEE VIS]]>https://revisit.dev/PR232/blog/2025/11/10/ieee-vis-award/2025-11-10T00:00:00.000ZWe’re happy to share that our paper on reVISit 2 received a Best Paper Award at IEEE VIS 2025. ReVISit 2 is an open framework for designing, deploying, and disseminating browser-based visualization studies across the full experiment life cycle. In this post, we summarize the contributions of the paper, describe the replication studies that demonstrate the system’s capabilities, reflect on feedback from users, and outline how reVISit can support more reproducible and expressive experimental research.
Running online visualization studies is now standard practice in VIS and HCI research. Yet the process remains fragmented: researchers stitch together survey tools, custom web code, logging scripts, analysis pipelines, and ad hoc debugging workflows. Users of reVISit already know this story: reVISit consolidates this ecosystem into a single open framework that supports the entire experiment life cycle – from design to dissemination.
To inform the academic community about the new developments in reVISit 2 – which we already described in these blogposts we wrote an academic paper about it.
In the paper, we first situate reVISit among existing study platforms. We compare it to commercial survey systems, domain-specific research tools, and library-based frameworks. While survey platforms excel at rapid deployment, they rarely support sophisticated interaction logging or fine-grained experimental control. Academic tools often address specific domains or slices of the workflow, but lack long-term maintainability or broad adoption.
ReVISit 2 is designed differently: it treats experiment design as programmable infrastructure. A JSON-based domain-specific language (DSL) models sequences, blocks, counterbalancing strategies, interruptions, skip logic, and dynamic control flow. On top of that, reVISitPy provides Python bindings that allow researchers to generate complex study configurations directly from notebooks. The result is a framework that emphasizes expressiveness, reproducibility, and ownership over one’s experimental stack.
We also describe technical advances in reVISit 2, including first-class Vega support, automated provenance tracking, participant replay, and improved debugging tools such as the study browser. These features aim to tighten feedback loops during piloting while preserving transparency during dissemination.
To demonstrate that these capabilities are not merely architectural, we conducted a series of replication studies.
Each study highlights a different capability of the system. In one, we implement adaptive and staircase-style designs to evaluate visualizations of correlations using dynamic sequencing logic, showing how complex control flow can be expressed directly in the study configuration. In another, we integrate think-aloud protocols by embedding audio recording and transcription into browser-based experiments, allowing researchers to capture reasoning during interaction rather than only after the fact. Finally, we demonstrate provenance tracking and replay by instrumenting interactive visualizations to capture detailed interaction histories, enabling fine-grained participant replay and qualitative analysis. reVISit 2 also provides deep linking to specific trials or moments in user studies, to aid in dissemination and transparency. For example, this link takes you to the exact state you see in the above image.
Across these replications, we recruited hundreds of participants and reproduced key findings from prior visualization studies. Just as importantly, the studies surfaced practical lessons about counterbalancing, recruitment logistics, and the realities of deploying sophisticated designs online. Together, they function both as validation and as a stress test of the framework. They also remain publicly accessible – complete with study configurations and data – serving as concrete, real-world examples that the community can inspect, reuse, and learn from.
We also interviewed experienced reVISit users to better understand how the system performs in practice. Several themes emerged:
Tighter development loops. Users appreciated the integrated development environment and the study browser, which makes it possible to jump directly to specific trials without stepping through an entire study.
Expressiveness over convenience. While the DSL requires programming knowledge, users valued the flexibility it affords – especially for mixed designs and adaptive sequencing.
Learning curve trade-offs. ReVISit inherits complexity from modern web tooling (e.g., React, TypeScript). This can be a barrier for less technical researchers, but it also enables deeper customization and extensibility.
Open infrastructure. The ability to fork studies, inspect core code, and maintain version stability was frequently cited as a strength, particularly for reproducibility.
Overall, feedback confirmed that reVISit is most effective for technically oriented research teams who need more than a survey builder.
We were very honored to receive an IEEE VIS Best Paper Award for this work. Zach Cutler presented the paper on the main stage in Vienna.
This recognition reflects years of iterative development, community feedback, tutorials, documentation work, and, most importantly, the researchers who have trusted reVISit in their own studies.
ReVISit 2 is not the endpoint. It is infrastructure for a research community that increasingly relies on sophisticated, reproducible, browser-based experiments. We look forward to continuing to build it together.
]]>The ReVISit Team<![CDATA[ReVISit v2.3: Databases, Screen Capture, Qualitative Analysis, Styling, and More!]]>https://revisit.dev/PR232/blog/2025/10/27/release-2.3/2025-10-27T00:00:00.000ZToday we're announcing several new reVISit features that were released in versions 2.1, 2.2 and 2.3. Major new features include an option to self-host data, remote screen-capture, a collaborative qualitative coding interface, and other features and improvements we're excited to share. For full details on the changes, look at the GitHub releases!
Supabase – A Self-Hosted Alternative to Firebase [Beta]
Maybe the biggest change is added support for Supabase, an open-source, self-hostable alternative to Firebase. Not all researchers can use products such as Firebase due to organizational or legal restrictions (GDPR, etc.). Supabase allows researchers with these restrictions to use reVISit inside an environment they fully control and own. Self-hosting is a bit more hands-on than using Firebase, so we make the setup as easy as possible by providing a docker-compose file to spin up Supabase on your own infrastructure. See Supabase setup guide to learn more.
Note: we ran a few studies with Supabase without issues, but we don’t have as much experience with it as we do with Firebase, so we're marking this feature as Beta until we have had more time to test with it.
Since its initial release, reVISit has included provenance tracking using Trrack. Provenance tracking can be very helpful for replaying and analyzing exactly how users interact with stimuli. However, it can also be tedious to setup – especially for complex visualizations with multiple elements to track, or when working with visualizations you don’t control (e.g. website components).
So today we’re introducing a flexible alternative for capturing user behavior: screen recording. In your study, you now can ask participants to grant browser permissions to record their screen – much like screen sharing in video conferencing software – with recordings stored in the reVISit infrastructure. This approach offers a much lower barrier to setup compared to provenance tracking and can be enabled directly through the reVISit config – no coding required!
The video shows an example screen recording session that also includes audio recording in a think-aloud protocol.
We’re finding screen recording to be extremely useful for early-stage piloting, debugging, as well as conducting full studies. In fact, if you have other reVISit capabilities enabled, such as think-aloud or provenance tracking, the screen recording syncs properly with them, making it powerful to analyze provenance as participants experience it. See the Screen recording setup guide to learn more. As with Supabase, we’re marking this feature as Beta until we have had more time to test with it.
We’ve added a qualitative analysis and coding platform to pair with our participant analysis and replay. This implementation is based on what we described in our CHI paper, with extensive design improvements and additional functionality. The analysis displays the transcribed text for a participant and task, and allows users to edit the transcriptions as well as create and apply codebooks to the transcribed audio.
The video shows an example of audio and video captured in a visualization drawing task, with the researcher applying codes to segments of the transcript.
Codes are available for download as part of the data export, making it easy to integrate qualitative coding into your analysis workflow.
We introduce new tools for customizing the look of the studies, making it easier to match your study’s design to your research goals. Whether you want to make your study polished or improve accessibility for participants, you can do it all directly with the styling options reVISit provides. Styling can be applied in multiple ways – from inline styles to external CSS files, depending on the level of customization you need.
Styles can be set at the global level for the whole study, at the component level for individual stimuli, or at the response (form element) level for specific questions.
With styles, you can get really into the design weeds. For instance, you can make reVISit feel deeply polished by adjusting the form style study, using clean layouts, different font sizes, and bordered responses. The video shows a study emulating Google Forms’ style:
You can try it out and look at various other examples here.
Advanced styling can be applied to make studies more engaging. For example, responses can enlarge text as participants start typing or change background when clicked. These interactions make study more engaging, providing a more user-friendly experience.
Styling allows researchers to create clean, accessible, and enjoyable studies without complicated steps. To see more styling examples, check out the styling documentation.
We added a new live monitor to the analytics interface. The monitor maintains a live connection to the database, and the moment participants answer a question, the heatmap and progress are updated automatically. This enables you to track participants’ progress without refreshing the page. For studies that have fixed-length trials, it displays all questions and marks answered questions in green. For studies that have dynamic components, it will show answered questions followed by a question mark since the total number of questions is not pre-determined. Note that the live monitor is currently only supported in Firebase-based studies.
The video shows the live monitor in action: in the left video, a participant is taking the study, and on the right, the live monitor updates automatically as the participant answers questions. We show both, a static and a dynamic study example.
A typical study goes through multiple stages, such as debugging by the developer, pre-pilots with friends and family; pilots with real participants and then finally the main study. In data analysis, you do want to keep these stages separate to avoid mixing pilot data with real data. To simplify this process, we introduce a new stage management feature.
You can now create stages and filter participants by their stage in the analysis interface. Stages are also prominently available in the data that you download from reVISit. By default, the study will be in the DEFAULT stage. Study designers can create new stages and give it a color, which will be assigned to new participants. In the table in the participant's view, we added a new column that displayed the stage name with corresponding color.
Advanced Control Over Study Flow With Dynamic Sequence Blocks
Dynamic blocks enable a study designer to build studies that require adaptive logic, i.e. the response to a question (e.g., right or wrong) can be used to determine the next (e.g., harder or easier) question.
We introduce a study summary page in the analysis interface. This provides a comprehensive overview of the study data in a single dashboard. The summary is organized in three sections – Overview Statistics, Component Statistics, and Response Statistics.
Overview Statistics show key information about the study, such as the total number of participants, study timeframe, average time spent, and the overall correctness of the study.
Component Statistics give details about each component of the study, such as number of participants, average time spent, and correctness.
Response Statistics show response level information including response type, answer options, and correctness.
This new summary page will help you understand participants' answers quickly. To learn more, please see the documentation.
The reVISit libraries are a growing collection of predefined study components and validated questionnaires for visualization and HCI research. With just a few lines of code, you can import them directly into your study.
We are excited to introduce several new libraries:
A virtual chinrest can be used to measure the distance between a participant and their monitor by detecting their blind spot. This reVISit library implementation was created by Sheng Long, and is the first community contribution!
In 2.1, we’ve added a video stimulus component, allowing participants to view video directly within the study. The video component supports both local videos or external video, with an option to force participants to watch the video which is useful to ensure that they watched the video before proceeding to answer the question. You can also add a timeline to the video to allow participants to skip to specific parts of the video. For more information please see the docs.
In 2.3, we’ve added the ranking widget, which allows participants to order or group items. This ranking component is useful for study designers who want to capture participants’ priorities, preferences, or how they categorize different items.
There are three types of ranking widgets that you can use depending on your study goals.
Ranking Sublist lets participants rank items in a single list, which is useful when you want them to focus on the relative order of their choices.
Ranking Categorical enables participants to rank items within high, medium, and low categories, which is useful when you want to understand which category an item belongs to rather than looking into the precise order.
Ranking Pairwise lets participants rank items by comparing them in high and low pairs, instead of ordering every item at once.
To learn more about how to use a ranking widget, refer to our docs.
The multiselect dropdown response became available in version 2.3. Previously, reVISit only offered a standard dropdown that allowed participants to select a single option. The multiselect dropdown enables participants to select multiple options. A dropdown response will automatically render as a multiselect dropdown if you define min or max selections in the configuration. Learn more about dropdown response in the docs.
In 2.2, we added a randomization feature for form elements and options to help reduce bias and improve the quality of study results. This feature allows individual questions, answer choices, or entire pages of questions to show up in random order for each participant, which is also recorded for replay. For more details on using randomization in studies, visit our docs.
Previously, participants were not allowed to go back in the study to change or correct their answers. In version 2.2, we added a previous button that allows participants to navigate back to earlier questions, if the study designer allows them to review and make corrections to their answers.
New Configuration Options and Analysis Interface Updates
ReVISit now allows you to set minimum screen size requirements to ensure participants have enough space to view study content and complete tasks. When a participant starts a study, reVISit checks their browser size and shows a warning page until the participant resizes their browser. If the requirements are not met within a time frame, the participant is automatically rejected. To learn more about how to apply screen size restriction to your study, please refer to the documentation.
Analysis Audio, Screen Recording and Transcript (Firebase) Download
ReVISit lets researchers capture and analyze participants' audio and screen recordings. Audio and transcripts, and screen recordings can now be downloaded for all participants or selected participants from the Participant View or Replay where previously you had to access these sources via Firebase.
]]>The ReVISit Team<![CDATA[ReVISit v2.0: Making your Studies even more Powerful!]]>https://revisit.dev/PR232/blog/2025/01/20/release-2.0/2025-01-20T00:00:00.000ZIt's the start of a new year and we're excited to announce the release of reVISit v2.0 — just in time for your VIS 2025 submissions! We've been working hard to bring you a new and improved version of reVISit, and we can't wait for you to try it out.
There are a lot of new features in this release, so let's dive in and take a look at what's new:
Ever wondered where your participants clicked when they completed your study? We've enabled participant replays, so you can now replay the interactions of participants in your studies during analysis. This enables you to see how participants are interacting with your study, either to discover issues in a pilot, or to actually analyze interaction behavior.
We've added compatibility for Vega and Vega-Lite visualizations as a component type, so you can now include these types of visualizations in your studies, leveraging the power of the reVISit platform. We also support tracking interactions in Vega visualizations, enabling you to inspect and analyze how participants used your stimulus with the aforementioned participant replay.
We've added support for recording participant audio, enabling you to run think-aloud studies—even in crowdsourced settings. This is a great way to gain insight into participants' thought processes and decision-making strategies and represents our latest effort to support qualitative research in reVISit. Audio recordings are automatically transcribed and part of your regular data download. You can even listen to the audio while watching the interactions of your participants play out.
Should we test for color blindness? What are our participants’ visualization literacy scores? How do participants rate the aesthetics of a visualization? We commonly ask these and similar questions, and often we use validated existing forms or methodologies to answer them. Re-implementing components is time-consuming and error prone. To address this, we've added support for libraries, so you can leverage prebuilt study components to create your own studies.
Libraries can save time and effort when creating studies, as you can reuse components that have already been created by others. You can also share your own components with the community by creating your own library and submitting a pull request.
Writing JSON can be difficult. Who wants to deal with a study config with 20,000 lines? To make it easier to create large and complex studies, we've implemented Python bindings for the reVISit spec, reVISitPy, that allow you to interact with reVISit programmatically. Here’s a basic example of how that works.
There are many promising things you can do with reVISitPy:
First, we implemented a widget that lets you run the study you created from inside the Jupyter notebook. Now you have a fast way of inspecting experiments, randomization, etc.
Next, you can run through the study and download the data you generated straight into your notebook, so that you can immediately see whether you’re actually collecting all the data you need, and even pilot your analysis!
Finally, with the Python bindings we also enable you to design arbitrarily complex studies from permutations. What does that mean? Say, you have a stimulus, such as a scatterplot, that can render any compatible dataset, and you want to test how good participants are at judging correlations. You might want to feed in hundreds of different datasets that you’re automatically generating. If you were to do this in the reVISit JSON spec, it would be very painful, because you have to create components for every single stimulus. In reVISitPy, it’s just a few lines of code. And of course, this isn’t limited to passing data into a component, you can permute over tasks, visual stimuli, or even phrasings of your questions, etc.
Improved User Interface: We've redesigned the user interface to make it more intuitive and user-friendly. You'll find it even more pleasant to use, with a cleaner and more modern look.
Forms have gotten some attention. See the demo and the documentation:
We introduced new matrix form elements, which are useful if you want to ask e.g., Likert questions for many different items.
Forms look nicer after an aesthetics overhaul.
We introduced dividers to section forms.
You can allow “I don’t know” as an option for most form elements.
You can allow “Other” options for checkboxes and radios.
You can choose horizontal and vertical layouts for checkboxes and radios.
You can now design trainings where participants can validate answers
Data export has improved, including things like participant numbers, clean time (time on task minus task where the browser tab was not active), etc.
These new features represent several months of work from the reVISit team, and we’re excited to share them with the community. We’re aiming to make reVISit more versatile, powerful, and easy to use. As always, we welcome your feedback and ideas for how we can support new directions for research in visualization and interactive systems. The best way to get in touch is to join our Slack Team!
We’re also ready to go on the road and meet you at your institution, or offer a virtual workshop! We recently visited Georgia Tech’s GVU center to give a hands-on overview of reVISit. Catch our upcoming workshop at CHI in Japan. Please reach out if you’re interested in learning more about reVISit or potentially hosting a workshop.
]]>The ReVISit Team<![CDATA[ReVISit v1.0: Taking Control of Your Online Studies!]]>https://revisit.dev/PR232/blog/2024/06/20/release-1.0/2024-06-20T00:00:00.000Z
You might have heard of reVISit before from our paper, or you might have seen a talk or participated in a meetup. But as of today, we’re excited to give the community a chance to run your own studies with reVISit with our 1.0 release – and CHI is just around the corner!
ReVISit is a software framework that enables you to assemble experimental stimuli and survey questions into an online user study.
One of the biggest time-saving features of reVISit is a JSON grammar, the reVISit Spec, used to describe the setup of your study.
Stimuli are contained in components and can be either markdown, images, web pages, React components, or survey questions.
The figure at the top shows the relationship of the reVISit Spec, the components, and how they are then compiled into a study.
Due to the different types of components, you can use reVISit for a diverse set of studies, spanning simple surveys, image-based perceptual experiments, and experiments evaluating complex interactive visualizations.
ReVISit is designed to accommodate sophisticated stimuli and study designs. Suppose you want to replicate the seminal Cleveland and McGill study. With reVISit you could implement a React-based set of visualizations (a bar chart, a stacked bar chart, a pie chart), and then pass parameters, such as the data, and the markers to highlight the marks, all via the study configuration.
Similarly, the reVISit spec enables designers to create controlled sequences defining in which order participants see stimuli. reVISit supports fixed, random, and Latin square designs that can be nested at various levels. For example, the overall study sequence (intro, training, experiment, survey) could be fixed. In the experiment arm, two conditions could use a Latin-square design. Within each condition, the experiment could randomly draw a small number of stimuli from a large stimuli pool while interspersing attention checks at random points and adding breaks.
The components and your study configuration are then used to assemble a web-based study. You can first look at your study on your local browser, and if you want to share it deploy it to the web server of your choice. We recommend and document deploying to GitHub pages, but any web server you have access to will do.
You can then use the online version to direct participants to your study. You can use crowdsourcing platforms such as Prolific, Mechanical Turk or LabintheWild, or you can simply send a link to participants that you have recruited in other ways.
A typical study will have response fields, such as a text field or a slider, to provide the response. Such form-based responses are tracked by reVISit by default and can be downloaded in either JSON or a tidy tabular format. Similarly, you can provide response data out of interactive stimuli. For example, if a task is to click on a specific bar in a bar chart, you can log which bars were clicked. ReVISit tracks a diverse set of browser window events such as mouse moves, clicks, scrolls, resizes, which are time-stamped and can hence be used for basic log file analysis.
ReVISit also supports advanced provenance tracking based on Trrack a provenance tracking library developed at our lab. If you instrument your study stimuli with Trrack, you can recreate every state of your interface of every single participant! This can be incredibly useful to understand nuances of user behavior, as well as to help you debug your stimuli by exploring what went wrong in a particular session. In a future release, reVISit will also allow you to dynamically browse these events and fully “re-hydrate” all participants experiments.
ReVISit is implemented as a (mostly) server-less application, meaning that you don’t have to run, secure, and maintain a server to use reVISit. The only exception to this is data storage, as obviously, the data of online participants has to be stored somewhere.
If you’re running a local study, you can get away without this – you can just download the data from your browser after a study is complete. For online studies, we use Google Firebase to store data.
Currently, setting up Firebase for a reVISit study might be the most challenging part of working with reVISit. On the plus side, Firebase is a tried-and-true system where you have full control over your data. You even have options to choose the locale of your server so that you are compliant with your country's regulations on data storage.
ReVISit is not meant to replace your usual data analysis approaches. Instead, it aims to make it easy to export data in the formats you might use in R, Python, or your analysis platform of choice.
ReVISit, however, does provide a basic analytics interface that is most useful for monitoring the progress of your study. You can also use reVISit to identify participants that didn’t appropriately complete the study and reject them, which is most useful if you want to ensure that you have appropriate numbers of participants in your Latin square design.
So, why would you use reVISit over other approaches to running your study, such as Qualtrics, Survey Monkey, or even a custom experiment interface?
First, reVISit is open source with all the benefits you have of using open source software: it’s free; you can extend it; you can contribute to improving it.
Second, the open source nature and our approach of forking reVISit for your own study and storing your data in your own Firebase means that you have full control over your study and the data. Once you have forked the study, it will remain accessible and unchanged for as long as you like.
Third, reVISit has dedicated modes for quickly navigating your study, and you can also turn off data collection. This is great for both, developing your study, but also sharing your study with reviewers and readers of your research. That means that readers can see exactly what your participants saw, and hence may trust your study more. They could also fork your study and run a replication of your study with minimal effort! You can check out an example study and the associated results.
I’m Intrigued, but Can I Trust it for my Experiment?
reVISit is new, and we know that it’s fraught to bet on a new project if you don’t know whether it actually works or whether it will be maintained down the line. But we hope we can convince you to trust us!
First, we currently have multiple years of funding to continue development of reVISit.
We’ve also run several successful studies, such as a study on guardrails against misinformation. Finally, we are committed to help you out if you run into issues! Join our slack team to get low-friction help, or write to us at contact@revisit.dev. We’re also happy to set up a meeting to answer any questions you may have; for example, to talk us through whether reVISit will work for your study design.
We’re grateful to all the community members who have shared their study needs and helped to make reVISit 1.0 a reality, and we’re looking forward to bringing the community exciting new features in the coming year. Future releases will include better debugging tools through study rehydration, a way to capture and code think-aloud data, and improved analysis capabilities. Depending on community feedback we're also interested in branching out to unconventional display devices (phones, AR/VR, etc.)
To take your first steps with reVISit, check out our getting started guide for instructions on how to install our software and to build a study.
Finally, if you are missing a feature or find a bug, let us know! Since reVISit is completely open source you could even submit a pull request!
