Authorship Verification
using Cloze Test and
Large Language Models
Mgr. Tomáš Foltýnek, Ph.D.
foltynek@fi.muni.cz
1
Contract Cheating
̶ Also known as ”assignment outsourcing”
̶ Students hire a third party to produce assignments for them
̶ Or use generative AI when not permitted
̶ A form of plagiarism
̶ Students submit someone else’s work
̶ Undetectable by current text-matching software tools
̶ Problem particularly in Anglo-Saxon educational setting
̶ Grades heavily depends on individual assignments (essays, homework)
̶ 3–8% of students commit contract cheating at least once
̶ 2/3 of them are repeat offenders
2
Cloze Test for Authorship Verification
̶ Cloze test proved to be a useful tool for
testing text comprehension. Some
universities use it during a disciplinary
procedure when a student is suspect from
submitting a work authored by someone or
something else (plagiarism, contract
cheating, unallowed use of generative AI).
Authors of the text are more likely to fill in
correct words.
̶ The project aims to find a method that
identifies words to be masked such that the
cloze test can reliably discriminate between
authors and non-authors. LLMs are trained
to predict the word in given context.
Previous experiments showed that nouns
that the model would not guess correctly
are good candidates.
3
Ultimate Goal: Authorship Verifier
̶ A web application to support the disciplinary procedure
̶ A student suspect of contract cheating is invited for interview
̶ His/her document is uploaded to the application
̶ The application automatically masks some words
̶ The student is asked to fill in the words (under supervision)
̶ The application gives a probability of authorship
̶ Not perfect, but much better then what is common now
4
TAČR Project
̶ Project partners
̶ Masaryk University, Faculty of Informatics
̶ Charles University, Faculty of Social Sciences
̶ Mendel University, Faculty of Business and Economics
̶ Application guarantor: IS MU
̶ Duration of the project: 09/2023 – 08/2026
̶ Budget for a student developer
̶ FTE gross salary 47 000 CZK
̶ Budget for part-time (60%), i.e. 24 hours per week
̶ 1 or 2 students
5
Current Status and Next Steps
̶ Current status
̶ Web application developed
̶ aver.pef.mendelu.cz
̶ A selection method
̶ Designed, implemented and tested
̶ Potential for improvements
̶ Drawback: Processing a document
takes too long (0,5 hour)
̶ Next steps
̶ To extend the existing project by conducting more experiments with LLMs
and users
̶ To improve existing method (better discriminate between authors and non-
authors)
6
Specific tasks
̶ Employ more language models to identify masked word (so far
only MT-5 was used)
̶ Experiment with probability of the word in given context (so far
only rank was used)
̶ Investigate the influence of language (English, German, etc.;
native / non-native)
̶ Investigate the influence of time (authors forget their text and
achieve lower scores)
7