Virtual Peer Chatbot Offers 24/7 Academic Support, Boosts Student Engagement

AI in Education: Virtual Peer Proves Effective in Pilot, Enhancing Academic Performance

Artificial Intelligence (AI) continues to transform various sectors, including education, with its ability to personalize learning experiences, enhance accessibility, and provide instant support. One significant example of this advancement is seen at Macquarie University in Sydney, where the AI chatbot "Virtual Peer" has demonstrated remarkable potential to improve students' academic performance.

In a pilot study conducted in October 2024 involving 1,400 psychology students, Virtual Peer enabled users to achieve a 9.45% increase in their grades within just two weeks before an exam. Developed in collaboration with Microsoft, the chatbot leverages Microsoft Azure’s access to OpenAI’s large language models (LLMs) to offer tailored assistance and guidance. The study controlled for participants' past academic performance, ensuring a fair assessment of the system’s effectiveness.

Macquarie University highlighted that students engage with AI tools more often than mid-career professionals. Along with other Australian universities, it has permitted the use of AI in certain assessments. However, findings from generative AI trials have presented mixed outcomes. For instance, a 2024 pilot of a Meta AI chatbot by Australia’s financial services regulator showed the technology performed worse than humans. Similarly, a six-month trial of Microsoft 365 Copilot by CSIRO revealed benefits in tasks like email coaching and meeting summaries but pointed to limitations in areas requiring deeper contextual understanding and workflow integration.

During its initial two-week trial, Virtual Peer exchanged over 20,000 messages with students, with 80% of interactions occurring outside regular university hours. This round-the-clock availability for both academic and administrative queries proved instrumental in its success. Lecturer Patrick Nalepka observed that the chatbot offered a safe and accessible platform for students to ask questions without judgment or fear of language barriers. A survey indicated that 72% of users would be very disappointed if the tool were discontinued, demonstrating its popularity and perceived value.

The Virtual Peer chatbot was designed to address academic and administrative queries using a curated knowledge base of lecture slides, transcripts, university policies, and Q&A materials. Unlike other large language models, Virtual Peer reduced errors or "hallucinations" by exclusively relying on verified data. Phil Laufenberg, Macquarie University's head of AI, explained that each response included references to source documents, enhancing transparency and clarity for students.

Moreover, Virtual Peer has benefited educators by providing anonymized chat logs. This feature allows lecturers to identify topics students find challenging, enabling them to tailor their teaching and revision sessions. Encouraged by the positive feedback from the pilot study, Macquarie University plans to expand the trial to over 3,000 students in 2025 and ultimately integrate the chatbot across hundreds of course units.

Microsoft noted that Virtual Peer’s accuracy still depended on the quality of its training data, with testing identifying minor gaps in documentation. While 84% of the chatbot’s responses to administrative queries were deemed equal to or better than human-generated responses, the remaining 16% of answers were linked to these gaps. To address this issue, Laufenberg shared that the onboarding process was refined to include standardized Q&A templates, allowing academics to provide clearer and more structured data. He acknowledged the complexities of developing AI at an enterprise level, emphasizing the valuable capabilities gained for future applications despite the system’s imperfections. Virtual Peer exemplifies how AI can enhance learning outcomes, streamline administrative support, and foster educational innovation, marking a step forward in modern education.

Editor’s Note:

The introduction of AI tools like Macquarie University’s Virtual Peer chatbot represents a significant step in revolutionizing education. By improving student performance and offering 24/7 academic and administrative support, AI has the potential to transform learning experiences. Virtual Peer’s success during its pilot trial highlights its effectiveness in addressing both academic queries and creating a safe, accessible environment for students to seek help. While the technology is not without its limitations, particularly in areas of contextual understanding, its proven benefits underline the growing role AI will play in education’s future. However, as AI continues to integrate into educational settings, the challenge of ensuring data quality and minimizing errors remains crucial. The minor gaps in Virtual Peer’s documentation, as identified by Microsoft, highlight the importance of rigorous training and clear communication between AI developers and educators.

Skoobuzz believes that the ongoing refinement of AI systems, such as the use of standardized Q&A templates to improve accuracy, shows that while the technology is evolving, it is essential to continue improving its effectiveness and reliability to ensure it delivers the best outcomes for students and educators alike.