Skip to main content Link Menu Expand (external link) Document Search Copy Copied
Table of contents

Assessment of XR-based Learning Experiences

Introduction

The assessment of XR-based learning experiences is necessary to understand not only whether participants appreciate an activity, but also whether the experience contributes to relevant learning outcomes. XR activities may influence conceptual understanding, practical skills, motivation, engagement, collaboration, confidence, and awareness of professional applications. Their evaluation should therefore consider multiple dimensions rather than relying exclusively on satisfaction or technology acceptance.

The assessment approach presented in this chapter was developed for two types of learning activities:

  1. Long-term co-design laboratories, conducted over several weeks and centred on the collaborative design and development of XR learning applications.
  2. Short intensive workshops, generally lasting three days and combining lectures, demonstrations, practical experimentation, teamwork, and final presentations.

Since these formats differ in duration, objectives, and participant involvement, they require related but distinct questionnaires. The proposed instruments combine closed-ended questions, which facilitate quantitative analysis, with open-ended questions, which help explain participants’ ratings and identify outcomes or problems not anticipated by the evaluators.

The questionnaires are intended as a reusable foundation. They can be adapted to the learning objectives, XR technology, target group, duration, and evaluation design of a specific activity.

Design of the Questionnaires

The questionnaire package has been designed according to the following principles:

  • Assessment of the learning experience, not only the technology. The evaluation focuses on the complete learning experience, including the learning activities, teamwork, facilitation, workload, and relationship with the intended learning outcomes. Questions about the XR system are included where relevant, but technical usability is not treated as a substitute for educational effectiveness.
  • Alignment with learning objectives. Questionnaire items should reflect the outcomes that the activity is expected to support. These may include:
    • deeper understanding of the subject;
    • retention and application of knowledge;
    • development of practical and technical skills;
    • problem-solving and design skills;
    • communication and teamwork;
    • confidence in addressing engineering problems;
    • awareness of professional roles and applications;
    • ability to design or evaluate XR-based learning experiences.

    This alignment should be performed before the learning activity begins. It should not be reconstructed only after results have been collected.

  • Combination of quantitative and qualitative evidence. Likert-scale questions permit comparisons among items, groups, learning formats, and editions of an activity. Open-ended questions provide the context needed to interpret the numerical results and can reveal unexpected benefits, barriers, or recommendations.
  • Assessment at appropriate points in time. For long-term activities, assessment should be repeated because participants’ perceptions may change as the work progresses. For short workshops, a pre-workshop questionnaire establishes expectations and prior experience, while a post-workshop questionnaire documents the experience and perceived outcomes.
  • Anonymity and proportionality. Only information required for the assessment should be collected. Responses should normally be anonymous or pseudonymous, and participants should be informed about the purpose of the questionnaire, the intended use of the results, and whether participation is voluntary.

Unless otherwise specified, statements concerning learning outcomes and the learning experience use the following scale:

  1. Strongly disagree
  2. Disagree
  3. Neither agree nor disagree
  4. Agree
  5. Strongly agree

An optional Not applicable / Unable to assess response should be provided when participants may not have had sufficient opportunity to evaluate an item. This response should be treated as missing data and should not be assigned a numerical value.

Questions about overall satisfaction use the following scale:

  1. Very dissatisfied
  2. Somewhat dissatisfied
  3. Neither satisfied nor dissatisfied
  4. Somewhat satisfied
  5. Very satisfied

Self-assessed competence can be measured using this scale:

  1. No competence
  2. Basic competence
  3. Intermediate competence
  4. Advanced competence
  5. Very advanced competence

Self-assessment reflects participants’ perceptions and should not be interpreted as an objective competence test. If competence development is a principal outcome, the questionnaire should be complemented by a knowledge test, practical task, rubric-based assessment, or observation.

Assessment of long-term co-design laboratories

A co-design laboratory involves sustained participation in the analysis, design, implementation, or evaluation of an XR-based learning application. Its assessment should address both the contribution of the laboratory to learning outcomes and the broader experience of collaborative development.

The questionnaires should be administered:

  • once after participants have gained sufficient experience of the laboratory;
  • optionally at an intermediate point and again at the end;
  • using the same core questions at each measurement point when longitudinal comparison is required.

An intermediate administration can identify problems while there is still time to modify objectives, responsibilities, supervision, or technical support.

This is a possible introductory text for participants:

This questionnaire examines your experience of participating in the co-design laboratory. Its purpose is to assess how the laboratory contributes to learning, skills development, teamwork, and confidence in addressing professional problems. The questionnaire is not an assessment of your individual performance. Please answer on the basis of your own experience. There are no right or wrong answers.

The questionnaire is split into two parts because they address different dimensions and can also be reused independently:

  1. Co-design Lab Questionnaire A: contribution to learning outcomes
  2. Co-design Lab Questionnaire B: learning experience and professional confidence

The two co-design lab questionnaires may be administered at the same time.

Co-design Lab Questionnaire A

Instruction: Indicate the extent to which you agree with each statement.

Scale: 1 = Strongly disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Strongly agree.

  1. The co-design laboratory helped me deepen my knowledge of the subject.
  2. The co-design laboratory helped me retain the knowledge acquired during the learning activity.
  3. The co-design laboratory helped me connect theoretical concepts with practical applications.
  4. The co-design laboratory helped me apply knowledge acquired in other courses or learning activities.
  5. The activities improved my understanding of XR technologies and their possible applications.
  6. The activities improved my ability to analyse an engineering or educational problem.
  7. The activities improved my ability to formulate possible solutions to a problem.
  8. The activities improved my ability to evaluate alternative design solutions.
  9. The activities improved my practical or technical skills.
  10. The activities improved my ability to work with digital models, XR applications, or related development tools.
  11. The activities improved my ability to consider the needs and characteristics of intended users.
  12. The activities improved my understanding of how learning objectives can be translated into an XR-based learning experience.
  13. The activities improved my ability to communicate technical ideas.
  14. The activities improved my ability to work collaboratively.
  15. The activities increased my confidence in addressing complex and open-ended tasks.

The following items should be included only when they match the laboratory objectives:

  1. The activities improved my ability to collect and interpret data from an XR learning experience.
  2. The activities improved my ability to evaluate the usability of an XR application.
  3. The activities improved my ability to evaluate the educational effectiveness of an XR application.
  4. The activities improved my ability to integrate physical and virtual systems.
  5. The activities improved my ability to document requirements, design decisions, and results.
  6. The activities improved my awareness of accessibility and inclusion in XR design.
  7. The activities improved my awareness of safety, privacy, and ethical issues associated with XR technologies.

Co-design Lab Questionnaire B

Instruction: Indicate the extent to which you agree with each statement.

Scale: 1 = Strongly disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Strongly agree.

  1. The objectives of the co-design laboratory were clear.
  2. The activities were relevant to the stated objectives.
  3. My responsibilities within the project were clear.
  4. I was able to make a meaningful contribution to the project.
  5. I had sufficient opportunities to express my ideas.
  6. The team made effective use of the different competences of its members.
  7. Communication among team members was effective.
  8. Collaboration with participants from other disciplines or institutions enriched the learning experience.
  9. The tutors provided appropriate guidance.
  10. The feedback received during the laboratory helped improve the work.
  11. The technical resources were adequate for the planned activities.
  12. The time available was adequate for completing the assigned work.
  13. The workload was appropriate in relation to the learning objectives.
  14. Technical difficulties were managed effectively.
  15. I was actively engaged throughout the laboratory.
  16. The laboratory increased my motivation to learn more about XR technologies.
  17. The laboratory increased my confidence in using XR-related tools.
  18. The laboratory increased my confidence in working on multidisciplinary projects.
  19. The laboratory increased my confidence in addressing professional engineering problems.
  20. The laboratory helped me better understand possible future professional roles.
  21. The laboratory helped me recognise how XR technologies may be used in my field.
  22. I would recommend participation in a similar co-design laboratory to other students.

Open-ended questions for co-design laboratories

  1. What were the most valuable aspects of the co-design laboratory?
  2. What knowledge or skills did you develop through the laboratory?
  3. Describe one contribution that you made to the project.
  4. What were the main difficulties encountered during the laboratory?
  5. How did the use or development of XR affect your learning experience?
  6. How did collaboration within the team affect your work and learning?
  7. What should be changed to improve the organisation of the laboratory?
  8. What additional technical or educational support would have been useful?
  9. In what contexts could the knowledge or skills acquired during the laboratory be applied?
  10. Is there anything else that you would like to report?

Assessment of short intensive workshops

Short workshops require an assessment design that distinguishes expectations from the actual experience. Two questionnaires are therefore recommended:

The pre-questionnaire documents participants’ backgrounds, previous experience, perceived competences, expectations, and concerns. The post-questionnaire examines satisfaction, engagement, learning, teamwork, organisation, and recommendations.

When possible, responses should be linked through anonymous participant codes. This permits paired analysis without collecting names or other direct identifiers.

Workshop pre-questionnaire

Introductory text for participants:

This questionnaire collects information about your previous experience, expectations, and perceived competences before the workshop. The results will be used to interpret the workshop assessment and improve future learning activities. The questionnaire is not a test, and individual responses will not be used to evaluate participants.

Participant background

  1. What is your field of study or professional background?
  2. What is your current level of study or professional experience?
  3. Which institution or organisation are you participating through?
  4. Have you previously participated in a multidisciplinary project?
    • No
    • Yes, once
    • Yes, more than once
  5. Have you previously worked in an international team?
    • No
    • Yes, once
    • Yes, more than once
  6. Have you previously used an XR system?
    • Never
    • Once
    • Occasionally
    • Frequently
  7. Which XR technologies have you previously used? Select all that apply.
    • Virtual reality
    • Augmented reality
    • Mixed reality
    • WebXR
    • Mobile AR
    • 360-degree video
    • Other
    • None
  8. Have you previously developed or configured an XR application?
    • No
    • Yes, with substantial guidance
    • Yes, with limited guidance
    • Yes, independently

Initial self-assessment of competence

Instruction: Rate your current level of competence in each area.

Scale: 1 = No competence; 2 = Basic; 3 = Intermediate; 4 = Advanced; 5 = Expert.

  1. Understanding the basic concepts of XR.
  2. Using VR or AR equipment.
  3. Navigating and interacting in a virtual environment.
  4. Creating or modifying three-dimensional content.
  5. Developing or configuring an XR application.
  6. Designing user interaction for an XR experience.
  7. Defining learning objectives.
  8. Designing a learning activity based on specified learning objectives.
  9. Evaluating the usability of a digital application.
  10. Evaluating the educational effectiveness of a learning activity.
  11. Working in a multidisciplinary team.
  12. Communicating technical ideas to people with different backgrounds.
  13. Solving open-ended engineering problems.
  14. Presenting project results.

Expectations

  1. What do you expect to learn during the workshop?
  2. Which knowledge or skills would you particularly like to develop?
  3. What do you expect from the practical activities?
  4. What do you expect from collaboration with other participants?
  5. What do you expect from the tutors and experts?
  6. How do you expect the workshop to contribute to your studies or professional development?
  7. Do you have any concerns about participating in the workshop?
  8. Are there any technical subjects or applications that you would particularly like to explore?
  9. List three to five adjectives that describe how you expect the workshop experience to be.

Optional pre-workshop agreement items

Scale: 1 = Strongly disagree to 5 = Strongly agree.

  1. I am motivated to participate in the workshop.
  2. I feel prepared to participate in the planned activities.
  3. I am confident that I can contribute to the team project.
  4. I expect the workshop to improve my understanding of XR.
  5. I expect the workshop to improve my practical skills.
  6. I expect collaboration with participants from different backgrounds to support my learning.
  7. I expect the workshop activities to be relevant to my studies or professional interests.

Workshop post-questionnaire

Introductory text for participants:

This questionnaire examines your experience of the workshop. The results will be used to assess the learning activities and improve future editions. Please consider the complete workshop, including presentations, demonstrations, practical activities, teamwork, and final presentations.

Overall satisfaction

  1. How satisfied were you with the workshop overall?

Scale: 1 = Very dissatisfied; 2 = Somewhat dissatisfied; 3 = Neither satisfied nor dissatisfied; 4 = Somewhat satisfied; 5 = Very satisfied.

  1. Please explain the main reason for your satisfaction rating.
  2. List three adjectives that describe your workshop experience.

The use of the same adjective task before and after the workshop enables comparison between anticipated and experienced perceptions. Adjectives can be normalised into common terms and analysed by frequency or thematic category.

Learning and relevance

Instruction: Indicate the extent to which you agree with each statement.

Scale: 1 = Strongly disagree to 5 = Strongly agree.

  1. The objectives of the workshop were clear.
  2. The activities were relevant to the workshop objectives.
  3. The workshop improved my understanding of XR technologies.
  4. The workshop improved my understanding of the possible applications of XR.
  5. The workshop helped me connect theoretical concepts with practical activities.
  6. The workshop improved my practical or technical skills.
  7. The workshop improved my ability to design an XR experience.
  8. The workshop improved my understanding of user-centred design.
  9. The workshop improved my ability to identify the needs of intended users.
  10. The workshop improved my ability to translate learning objectives into an XR activity.
  11. The workshop improved my problem-solving skills.
  12. The workshop increased my confidence in working with XR technologies.
  13. The workshop was relevant to my studies or professional development.
  14. I expect to apply something learned during the workshop in future activities.

Engagement

Instruction: Rate each statement in relation to your experience during the workshop.

Scale: 1 = Strongly disagree to 5 = Strongly agree.

  1. I was actively involved in the workshop activities.
  2. I was able to maintain attention during the activities.
  3. I was motivated to complete the assigned tasks.
  4. I felt capable of contributing to the activities.
  5. I was satisfied with my contribution.
  6. I had sufficient control over the tasks assigned to me.
  7. I was able to perform the activities successfully.
  8. The level of challenge was appropriate.
  9. The workshop stimulated my curiosity about XR.
  10. I would like to continue learning about the topics addressed.

Teamwork and collaboration

Instruction: Indicate the extent to which you agree with each statement.

Scale: 1 = Strongly disagree to 5 = Strongly agree.

  1. The team understood the assigned challenge.
  2. Responsibilities were distributed effectively within the team.
  3. I was able to make a meaningful contribution.
  4. Communication within the team was effective.
  5. Team members listened to different perspectives.
  6. The multidisciplinary composition of the team improved the proposed solution.
  7. Collaboration helped me learn from other participants.
  8. The team managed its available time effectively.
  9. The team received adequate support from tutors.
  10. I was satisfied with the outcome produced by the team.

If a single overall collaboration question is needed, the following item can be used:

How would you rate collaboration within your team?

  1. Very poor
  2. Poor
  3. Fair
  4. Good
  5. Excellent

Organisation and support

Scale: 1 = Strongly disagree to 5 = Strongly agree.

  1. Information provided before the workshop was adequate.
  2. The programme and schedule were clear.
  3. The balance among lectures, demonstrations, and practical work was appropriate.
  4. The time allocated to practical activities was adequate.
  5. The time allocated to team project work was adequate.
  6. The technical equipment and software were adequate.
  7. The learning materials and instructions were clear.
  8. The tutors provided appropriate support.
  9. The facilities supported the planned activities.
  10. The overall workload was appropriate.

Post-workshop self-assessment of competence

For pre/post comparison, repeat the same competence questions used in the pre-workshop questionnaire:

  1. Understanding the basic concepts of XR.
  2. Using VR or AR equipment.
  3. Navigating and interacting in a virtual environment.
  4. Creating or modifying three-dimensional content.
  5. Developing or configuring an XR application.
  6. Designing user interaction for an XR experience.
  7. Defining learning objectives.
  8. Designing a learning activity based on specified learning objectives.
  9. Evaluating the usability of a digital application.
  10. Evaluating the educational effectiveness of a learning activity.
  11. Working in a multidisciplinary team.
  12. Communicating technical ideas to people with different backgrounds.
  13. Solving open-ended engineering problems.
  14. Presenting project results.

The competence scale must remain unchanged between the pre-workshop and post-workshop questionnaires.

Open-ended post-workshop questions

  1. What were the most valuable aspects of the workshop?
  2. What were the most important things that you learned?
  3. Which activity contributed most to your learning, and why?
  4. How did the practical use of XR affect your understanding of the topic?
  5. How did teamwork affect your learning experience?
  6. What difficulties did you encounter?
  7. Which aspects of the organisation should be improved?
  8. Was the time available sufficient? Please explain your answer.
  9. Which topics or technologies should be included in a future workshop?
  10. Which additional practical activities would you recommend?
  11. How might you apply the knowledge or skills acquired during the workshop?
  12. Is there anything else that you would like to report?

Administration guidance

Before completing the questionnaire, participants should receive information about the purpose of the assessment, the voluntary nature of participation, the data collected, the intended use and retention of the responses, and the contact point for questions or withdrawal requests. Where questionnaire data are used for research or publication, the assessment must follow the applicable institutional ethics procedures and data-protection requirements. Demographic information should be limited to variables that are necessary for the analysis, particularly when the participant group is small.

Anonymous response codes

For pre/post comparisons, participants can create a non-identifying code that they can reproduce at both measurement points. The code-generation method should avoid combinations that could readily reveal identity in a small group.

The same anonymous code should be used only if paired analysis is planned. Otherwise, fully anonymous questionnaires may be preferable.

Questionnaire length

As a practical guideline:

  • a pre-workshop questionnaire should normally require no more than 5–10 minutes;
  • a post-workshop questionnaire should normally require no more than 10–15 minutes;
  • a co-design-lab questionnaire should normally require no more than 10–15 minutes.

Longer instruments may reduce completion rates and lead to superficial responses, particularly when administered at the end of an intensive activity.

Core and optional items

A stable group of core questions should be retained across different editions. Optional questions can then be added to reflect specific technologies, learning objectives, or activities.

This separation makes it possible to compare results over time without preventing adaptation to a particular context.

The following contextual information should be recorded for each administration:

  • name and type of learning activity;
  • activity dates and duration;
  • measurement point;
  • number of invited participants;
  • number of respondents;
  • response rate;
  • participant group;
  • questionnaire version;
  • language of administration;
  • XR hardware and software used;
  • main learning objectives;
  • whether responses are anonymous, pseudonymous, or identifiable.

These data are needed to interpret the results and support comparison or reuse.