Trelis/modernbert-embed-base-touch-rugby-ft

SentenceTransformer based on nomic-ai/modernbert-embed-base

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the touch-rugby-modernbert-pairs dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • touch-rugby-modernbert-pairs

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Trelis/modernbert-embed-base-touch-rugby-ft")
# Run inference
sentences = [
    'Is there a time-out for injuries during a standard Touch Rugby match?',
    '4.9\tReferees and players may wear spectacles or sunglasses provided they are safe \nand securely attached.4.10\tReferees and players may wear sport monitoring equipment and medical \nsupports such as knee or ankle braces provided, at the sole discretion of \ncompetition’s controlling body, the items are not dangerous.5\u2002 Team Composition  \n5.1\tA Team consists of a maximum of 14 players, no more than six (6) of whom are \nallowed on the field at any time.FIT Playing Rules - 5th Edition\n6\nCOPYRIGHT © Touch Football Australia 2020\nRuling = A Penalty awarded to the non-offending Team at the time the offence is identified \nseven (7) metres infield on the Halfway Line or the position of the ball, whichever is the \ngreater Advantage.5.2\tA Team must have a minimum of four (4) players on the field for a match to \ncommence or continue, except during a Drop-Off.',
    '10.10\tIf a player in Possession intentionally makes a Touch on an Offside defender \nwho is making every effort to retire and remain out of play, the Touch counts.FIT Playing Rules - 5th Edition\nCOPYRIGHT © Touch Football Australia 2020\n9\n10.11\tIf a Touch is made on a player in Possession while the player is juggling the ball \nin an attempt to maintain control of it, the Touch counts if the attacking player \nfollowing the Touch retains Possession.10.12\tIf a player in Possession is Touched and subsequently makes contact with \neither the Sideline, a field marker or the ground outside the Field of Play, the \nTouch counts and play continues with a Rollball at the Mark where the Touch \noccurred.10.13\tWhen a player from the Defending Team enters its defensive Seven Metre Zone, \nthe Defending Team must move Forward at a reasonable pace until a Touch is \nImminent or made.Ruling = A Penalty to the Attacking Team at the point of the Infringement.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

touch-rugby-modernbert-pairs

• Dataset: touch-rugby-modernbert-pairs at 7cb0ae2
• Size: 305 training samples
• Columns: question and related_chunk
• Approximate statistics based on the first 305 samples:

  	question	related_chunk
  type	string	string
  details	min: 10 tokens mean: 18.73 tokens max: 36 tokens	min: 147 tokens mean: 231.2 tokens max: 319 tokens

• Samples:

  question	related_chunk
  What is defined as 'Forward' in the context of Touch Rugby?	Referee The match official(s) appointed to make Rulings during the conduct of a match.Rollball The act of bringing the ball into play following a Touch or a Change of Possession.Ruck/Rollball Area The area, not exceeding one (1) metre in distance, between the player performing a Rollball and the Half.Ruling The decision made by a Referee as a result of particular circumstance and may result in a Play On, a Tap Penalty, a discipline option, Change of Possession or a Try.Seven Metre Zone The area between the seven (7) metre line and the Try Line.See Appendix 1.Sidelines The side boundaries of the Field of Play.See Appendix 1.Sin Bin A player sent to the Sin-Bin Area for a period of four (4) completed Possessions.The player is counted as a player on the Field of Play and cannot be replaced or Interchanged.
  What is the numerical difference between the teams on the field of play during a Drop-Off if a player has been sent to the sin bin?	24.5 For the avoidance of doubt for clauses 24.3 and 24.4 the non-offending Team will retain a numerical advantage on the Field of Play during the Drop-Off.25  Match Officials 25.1 The Referee is the sole judge on all match related matters inside the Perimeter for the Duration of a match, has jurisdiction over all players, coaches and officials and is required to: 25.1.1 Inspect the Field of Play, Line Markings and Markers prior to the commencement of the Match to ensure the safety of all participants.25.1.2 Adjudicate on the Rules of the game; 25.1.3 Impose any sanction necessary to control the match; 25.1.4 Award Tries and record the progressive score; 25.1.5 Maintain a count of Touches during each Possession; 25.1.6 Award Penalties for Infringements against the Rules; and 25.1.7 Report to the relevant competition administration any Sin Bins, Dismissals or injuries to any participant sustained during a Match.
  What happens if neither team is leading after two minutes of play in a Drop-Off?	24.1.5 Should neither Team be leading at the expiration of two (2) minutes, a signal is given and the match will pause at the next Touch or Dead Ball.Each Team will then remove another player from the Field of Play.24.1.6 The Match will recommence immediately after the players have left the field at the same place where it paused (i.e.the Team retains Possession at the designated number of Touches, or at Change of Possession due to some Infringement or the sixth Touch) and the Match will continue until a Try is scored.24.1.7 There is no time off during the Drop-Off and the clock does not stop at the two (2) minute interval.24.1.8 Substitution during the Drop-Off is permitted in accordance with normal Interchange Rules.24.2 Mixed gender Teams may have no more than (2) males on the field during the Drop-Off.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

touch-rugby-modernbert-pairs

• Dataset: touch-rugby-modernbert-pairs at 7cb0ae2
• Size: 305 evaluation samples
• Columns: question and related_chunk
• Approximate statistics based on the first 305 samples:

  	question	related_chunk
  type	string	string
  details	min: 13 tokens mean: 17.61 tokens max: 32 tokens	min: 147 tokens mean: 230.39 tokens max: 319 tokens

• Samples:

  question	related_chunk
  What is the penalty for an attacking player obstructing a defender in Touch Rugby?	Ruling = A Penalty to the Attacking Team at the point of the Infringement or on the seven (7) metre line directly Forward of the Infringement.15.4 When a Rollball occurs within Defending Team’s Seven Metre Zone or a Penalty Tap within ten (10) metres of the Defending Team’s Try Line, all players from the Defending Team must have both feet on or behind their Try Line and no other part of the body in contact with the ground Forward of their Try Line.Ruling = A Penalty to the Attacking Team at the seven (7) metre line directly Forward of the point of the Infringement.15.5 After effecting the Touch, the defending player must retire the required seven (7) metres or to the Defending Try Line as indicated by the Referee without interfering with the Attacking Team.Ruling = A Penalty to the Attacking Team ten (10) metres Forward of the Infringement or if on the Defensive Try Line, on the seven (7) metre line.
  When must a player perform a Rollball seven metres in-field?	13.5 A player may only perform a Rollball at the Mark under the following circumstances: 13.5.1 when a Touch has been made; or 13.5.2 when Possession changes following the sixth Touch; or 13.5.3 when Possession changes due to the ball being dropped or passed and goes to the ground; or 13.5.4 when Possession changes due to an Infringement by an attacking player at a Penalty, a Tap or a Rollball; or FIT Playing Rules - 5th Edition COPYRIGHT © Touch Football Australia 2020 11 13.5.5 when Possession changes after the Half is Touched or when the Half places the ball on or over the Try Line; or 13.5.6 in replacement of a Penalty Tap; or 13.5.7 when so directed by the Referee.13.6 A player is to perform a Rollball seven (7) metres in-field under the following circumstances: 13.6.1 when a Change of Possession takes place due to a player in Possession making contact with the Sideline or any ground outside the Field of Play, prior to a Touch being made; or 13.6.2 when the ball not in Poss...
  What is the ruling if a player uses excessive force when making a touch?	FIT Playing Rules - 5th Edition 8 COPYRIGHT © Touch Football Australia 2020 9.6 If a player mishandles the ball and even if in an effort to gain control, the ball is accidentally knocked Forward into any other Player, a Change of Possession results.10  The Touch 10.1 A Touch may be made by either a defending player or a player in Possession.10.2 A defending player may not claim a Touch if contact has not been made.If a player claims a Touch has been made, but the Referee is unsure the Touch will count.Ruling = A Penalty to the Attacking Team at the point of the Infringement and the offending player sent to the Sin Bin.10.3 Players of both Defending and Attacking Teams are to use the minimum force necessary to make a Touch.Players must ensure that the method employed in making a Touch does not pose an unnecessary risk to player safety.Ruling = A Penalty to the non-offending Team at the point of the Infringement.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_ratio: 0.3

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.3
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.2222	2	2.7671	nan
0.4444	4	0.0	nan
0.6667	6	0.0	nan
0.8889	8	0.0	nan

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 3.3.1
• Transformers: 4.48.0
• PyTorch: 2.5.1
• Accelerate: 1.3.0
• Datasets: 2.17.1
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}