Student Academic Consultation Chatbot Using Meta AI Large Language Models and Retrieval-Augmented Generation

Sari

Academic consultation is an important service for students in obtaining information related to academic regulations, procedures, and requirements. However, the consultation process, which is still carried out manually, often causes delays in the delivery of information and limited access, especially when students need answers quickly. Therefore, a system is needed that is capable of providing academic consultation services automatically and based on official documents. This study aims to design and build a student academic consultation chatbot using Large Language Model (LLM) technology and Retrieval Augmented Generation (RAG) architecture. The methods used include calling up Academic Guidelines documents, splitting text into several parts (text splitting), creating embeddings using the HuggingFace all-MiniLM-L12-v2 model, and storing embeddings in a vector database. Next, the system performs a relevant document search process using a retriever and utilizes the LLaMA 3.1-8B-Instant model to generate answers based on the context found. The chatbot's performance was evaluated using ROUGE metrics, including ROUGE-1, ROUGE-2, and ROUGE-L, with measurements of precision, recall, and F1-score. The evaluation results showed that the chatbot was able to provide relevant answers in accordance with academic documents. The average evaluation scores obtained were precision of 47,57%, recall of 67,85%, and F1-score of 53,36%. The higher recall score indicates that the system is quite good at covering reference information, although the accuracy of word selection can still be improved.