Model Card for TriboBERT

TriboBERT is a domain-specific pre-trained language model built upon MatSciBERT and continually pre-trained on a large corpus of tribology literature via masked language modeling (MLM). It is designed as a reusable infrastructure for tribological text understanding—enabling researchers to extract semantic representations from unstructured literature without manual feature engineering.

Model Details

Model Description

TriboBERT is a BERT-base architecture (≈110M parameters) adapted to the tribology domain through continual pre-training on a large-scale corpus of tribology-related scientific literature. The model is intended to serve as a foundational language model for a wide range of tribology-related natural language processing tasks, including feature extraction, fine-tuning for named entity recognition and text classification, and semantic search over tribological literature.

This model is built upon MatSciBERT[reference:0], a BERT model pre-trained on materials science literature. Through domain-adaptive pre-training on tribology-specific texts, TriboBERT acquires deep understanding of tribological terminology, material systems, and experimental contexts, enabling it to outperform general-domain baselines on downstream tribological tasks.

• Developed by: Wenhao He (Lanzhou Institute of Chemical Physics, Chinese Academy of Sciences)
• Funded by [optional]: Strategic Priority Research Program of the Chinese Academy of Sciences (XDB 0470203); National Natural Science Foundation of China (12302128); West Light Foundation of CAS (xbzg-zdsys-202305); Key Cultivation Projects of LICP (KCP155B04)
• Shared by [optional]: Wenhao He
• Model type: Masked Language Model (BERT-base architecture)
• Language(s) (NLP): English (scientific text)
• License: Apache 2.0
• Finetuned from model [optional]: m3rg-iitd/matscibert

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

TriboBERT can be used directly for extracting semantic embeddings from tribological text. The [CLS] token embedding from the final hidden layer (768-dimensional) can be fed into downstream regression or classification models for property prediction, as demonstrated in our DLC friction coefficient prediction task.

Direct Use

from transformers import BertForMaskedLM, BertTokenizer
import torch

model = BertForMaskedLM.from_pretrained("your-username/tribobert")
tokenizer = BertTokenizer.from_pretrained("your-username/tribobert")

text = "A Si-DLC coating was deposited by PACVD on an AISI 52100 substrate."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512, padding=True)
with torch.no_grad():
    outputs = model(**inputs, output_hidden_states=True)
    cls_embedding = outputs.hidden_states[-1][:, 0, :]  # [CLS] token

### Downstream Use [optional]

The model can be fine-tuned for task-specific applications such as:

Named Entity Recognition (NER): Extraction of lubricant materials, compositions, and testing conditions

Text Classification: Categorization of tribological research domains

Relation Extraction: Identifying relationships between compositions and properties

Semantic Search: Similarity-based retrieval of tribological literature


### Out-of-Scope Use

TriboBERT is a domain-specific language model for tribology and materials science. It is not intended for general-purpose NLP tasks unrelated to tribology, nor for generative tasks such as open-ended dialogue or creative writing. The model should not be used for making safety-critical decisions without appropriate validation.

## Bias, Risks, and Limitations

Domain specificity: The model's vocabulary and semantic understanding are specialized for tribology and may not generalize well to other scientific or non-scientific domains.

Training data bias: The pre-training corpus primarily consists of English-language scientific articles from the Web of Science, which may reflect publication biases toward certain materials, regions, or research topics.

Task-specific fine-tuning required: While the model provides strong domain-adapted representations, most practical applications require additional fine-tuning on task-specific labeled corpora.

No factual verification: The model does not verify factual correctness of statements and should not be used as a standalone knowledge base without human oversight.

### Recommendations

Users should:

Fine-tune the model on task-specific labeled data for optimal performance

Validate predictions against experimental or known results when used for property prediction

Be aware of potential biases in the training data and interpret results accordingly


## How to Get Started with the Model

Installation
pip install transformers torch


#Load the Model and Tokenizer

from transformers import BertForMaskedLM, BertTokenizer

model = BertForMaskedLM.from_pretrained("your-username/tribobert")
tokenizer = BertTokenizer.from_pretrained("your-username/tribobert")


#Extract Feature Embeddings
import torch

text = "A Si-DLC coating was deposited by PACVD on an AISI 52100 substrate."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512, padding=True)
with torch.no_grad():
    outputs = model(**inputs, output_hidden_states=True)
    cls_embedding = outputs.hidden_states[-1][:, 0, :]  # [CLS] token
print(cls_embedding.shape)  # torch.Size([1, 768])


#Fine-tune for Downstream Tasks
from transformers import BertForSequenceClassification, Trainer, TrainingArguments

model = BertForSequenceClassification.from_pretrained("your-username/tribobert", num_labels=2)
training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)
trainer.train()



## Training Details

### Training Data

The pre-training corpus was compiled from the Web of Science Core Collection using the query:

TS = ((friction OR wear OR lubricat* OR tribology) AND (surface OR contact OR interface))

After cleaning and PDF extraction with GROBID, the final corpus consists of:

101,673 full-text articles (published before 2020)

197,661 abstracts (as of March 2026)

The dataset covers a wide range of tribology topics including solid lubrication, liquid lubricants, coatings, wear mechanisms, and contact mechanics.


### Training Procedure

The model was initialized from the pre-trained MatSciBERT weights and continually pre-trained using the masked language modeling (MLM) objective.

Preprocessing
Text extraction: Full-text PDFs were parsed using GROBID, a machine learning library that extracts structured TEI-XML containing titles, authors, abstracts, body text, and references.

Tokenization: BERT tokenizer with a maximum sequence length of 512 tokens.

Sliding window: A stride of 256 tokens was used to process longer paragraphs.

Masking: 15% of tokens were masked following the standard BERT MLM procedure (80% [MASK], 10% random token, 10% unchanged).




#### Training Hyperparameters

Parameter	Value
Optimizer	AdamW
Learning rate	2×10⁻⁵
Weight decay	1×10⁻²
Learning rate schedule	Linear decay with 6% warmup
Effective batch size	32 (batch size 4 × gradient accumulation 8)
Masking probability	15%
Max sequence length	512 tokens
Sliding window stride	256 tokens
Precision	FP16 mixed precision (torch.amp)
Epochs	3

#### Speeds, Sizes, Times [optional]

Hardware: Single NVIDIA A100 80GB GPU

Total training time: ~72 hours for 3 epochs

Model size: ~110 million parameters (~440 MB in pytorch_model.bin)

Training loss progression: MLM loss decreased from ~1.65 to ~1.26; perplexity (PPL) decreased from ~5.20 to ~3.52 (≈32% relative reduction)



## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

TriboBERT was evaluated on three diagnostic tasks:

Abbreviation–full-name alignment: 20 representative tribological abbreviations (e.g., DLC, MoS₂, PTFE, ZDDP)

Unsupervised semantic clustering: 197,661 tribology-related abstracts

Friction coefficient (COF) prediction: 193 DLC coating entries from the Sedlacek et al. database


#### Metrics

Task 1: Top‑1 matching accuracy (cosine similarity of [CLS] embeddings)

Task 2: Qualitative cluster coherence via UMAP + HDBSCAN, characterized by c‑TF‑IDF keywords

Task 3: Coefficient of determination (R²) and Mean Absolute Error (MAE) via 5‑fold cross-validation



### Results

Task	TriboBERT	MatSciBERT	SciBERT	BERT
Abbreviation alignment (Top‑1)	90%	10%	10%	5%
COF prediction (R²)	0.92 ± 0.03	0.88 ± 0.04	0.87 ± 0.03	0.90 ± 0.03
COF prediction (MAE)	0.016 ± 0.004	0.023 ± 0.004	0.022 ± 0.004	0.021 ± 0.004
TriboBERT consistently outperforms all baselines on these diagnostic tasks, demonstrating the effectiveness of domain-adaptive pre-training for tribological text understanding.



#### Summary

## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

TriboBERT follows the BERT-base architecture:

Layers: 12 transformer encoder layers

Hidden size: 768

Attention heads: 12

Parameters: ~110 million

Training objective: Masked Language Modeling (MLM)


### Compute Infrastructure

#### Hardware

GPU: Single NVIDIA A100 80GB

CPU: 92 cores (for parallel preprocessing)

Memory: Sufficient for 512-token sequences with batch size 4

#### Software

Framework: PyTorch + Hugging Face Transformers

Tokenizer: BERT tokenizer (bert-base-uncased)

Preprocessing: Hugging Face datasets library with multiprocessing (80 workers)

Mixed precision: torch.amp (FP16)


## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

MLM (Masked Language Modeling): A pre-training objective where a percentage of input tokens are masked and the model is trained to predict them.

[CLS] token: A special token prepended to each input sequence; its final hidden state serves as an aggregate representation of the entire sequence.

Perplexity (PPL): exp(loss), a measure of how well a language model predicts a sample; lower perplexity indicates better predictive performance.

GROBID: A machine learning library for extracting structured information from scholarly PDF documents.

Domain-adaptive pre-training: Continued pre-training of a general-domain model on domain-specific corpora to adapt its representations.



## More Information [optional]

For questions or issues, please contact the authors (Wenhao He, Hewh@licp.cas.cn).

## Model Card Authors [optional]

Wenhao He

## Model Card Contact

Wenhao He, Hewh@licp.cas.cn