josh-oo/german-gpt2-easy

Easy German GPT2 Model

A language model for german easy language ("leichte Sprache") based on German GPT-2 model

Model Details

Initialized using the weights of German GPT-2 model.
Then fine-tuned for one epoch on "leichte Sprache" corpora consisting of:

• encyclopedia like data
• news like data

Hyperparamters used for fine-tuning:

• tokenizer:

  • max_length: 1024 (but trained with dynamic length, using the collator functions 'pad_to_multiple_of=8')
  • stride: 64
  • return_overflowing_tokens=True

• training arguments

  • num_train_epochs=1
  • learning_rate=1e-3
  • weight_decay=0.01
  • per_device_train_batch_size=4
  • gradient_accumulation_steps=4
  • warmup_steps=200
  • fp16=True

→ 25112 training items, trained on google colab GPU (30 min)

Evaluation results

The perplexity value is calculated based on an unseen dataset containing manually aligned standard german and "leichte Sprache" texts.
For calculation, the method described in this tutorial was used with the following values:

• max_length = 512
• stride = 256

For comparison: running the modified function on this example gives us a perplexity score of 18.2551

Model	Perplexity "leichte Sprache" (Easy MDR News))	Perplexity standard german (Standard MDR News)
German GPT-2 model	23.8257	24.0301
our model	17.3053	48.6314