hafsteinn/skemman-detector-en-roberta-large

Skemman AI-text detector: RoBERTa-large, English

A binary classifier that scores whether an English university thesis abstract resembles an abstract generated by a large language model. It is one of the detectors in the paper Detecting AI-associated writing in Icelandic and English university thesis abstracts (under review), which applies them to abstracts deposited in Skemman, the repository of Icelandic universities.

In the paper it is a member of the English ensemble, whose score is the mean of the members' scores.

This is the original detector, trained and applied on text with its line breaks. Generated training abstracts contained paragraph breaks far more often than deposited abstracts, and some detectors relied on this cue, so the paper's main results use the variant trained without line breaks (hafsteinn/skemman-detector-ws-en-roberta-large). This detector is compared with it in Appendix D.

Score

The score is the difference of the two classifier logits, logit[1] - logit[0]: the log-odds of the generated class against the historical class, computed on the first 512 tokens.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "hafsteinn/skemman-detector-en-roberta-large"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

text = "..."
with torch.no_grad():
    logits = model(**tokenizer(text, truncation=True, max_length=512, return_tensors="pt")).logits[0]
score = (logits[1] - logits[0]).item()

Higher scores are more similar to generated text. The score is a classification score, not a calibrated probability that a text was written with AI.

Training data

Abstracts from 690 theses deposited in Skemman in 2019–2021, paired with alternative abstracts that 36 models from OpenAI, Anthropic and Google generated from the corresponding thesis bodies. Originals and generated alternatives from the same thesis were kept in one split; 77 further theses were used for validation. The training data are not redistributed.

Fine-tuned from FacebookAI/roberta-large as a binary sequence classifier on 1,048 abstracts (358 historical and 690 generated): AdamW, peak learning rate 1e-05 with linear warm-up and decay, weight decay 0.01, class-weighted cross-entropy, batch size 16, seed 0, 4 epochs, keeping the epoch with the highest validation AUROC.

Evaluation

On the labelled test set from 230 theses deposited from January to September 2022 (230 generated and 110 historical English abstracts), the test AUROC is 0.997 (95% thesis-bootstrap interval 0.990–1.000).

In the paper, this model's 1% false-positive threshold is 2.135 on the score scale. It was fitted to historical abstracts from 2019 to September 2022 outside the 997 theses used to develop the detectors. The English ensemble threshold is 0.777.

Intended use and limitations

• The detector is meant for corpus-level research on AI-associated writing, such as changes over time in a repository. A score shows that a text resembles generated text. It does not show how a text was written or whether any assistance was permitted or declared, and it should not be used as evidence about an individual student.
• It was evaluated on fully generated abstracts. Edited, translated, paraphrased or mixed human–AI texts were not evaluated, nor was deliberate evasion.
• It was trained and tested on thesis abstracts in Skemman; other genres and repositories are untested.
• Inputs longer than 512 tokens are truncated.
• This tokenizer encodes line breaks, and generated training abstracts contained paragraph breaks far more often than historical ones, so the model can use formatting as a cue. The skemman-detector-ws-en-roberta-large variant was trained without line breaks.

License

This detector is released under the Creative Commons Attribution 4.0 International license (CC BY 4.0). Its base model, FacebookAI/roberta-large, is released under MIT.