---
library_name: transformers
license: other
base_model: Synthyra/ESMplusplus_large
base_model_relation: finetune
pipeline_tag: feature-extraction
tags:
- protein-language-model
- protein-sequence
- capsid
- ESMCapsid
- ESMplusplus
- masked-language-modeling
- custom_code
---

# ESMCapsid-C

> Built with ESM.

ESMCapsid-C is a continued masked-language-model fine-tune of [`Synthyra/ESMplusplus_large`](https://huggingface.co/Synthyra/ESMplusplus_large) for the representation stage of the ESMCapsid workflow. The repository includes the model weights, tokenizer, configuration, and custom Transformers implementation required for loading the model.

## About

ESM++ Large provides a Transformers-compatible implementation of ESMC 600M. ESMCapsid-C was initialized from that checkpoint and continued with masked-language-model training. In the released ESMCapsid pipeline, sequences passing the ESMCapsid-S screen are encoded with Layer 35 and attention-mask mean pooling.

| Property | Value |
| --- | --- |
| Base model | `Synthyra/ESMplusplus_large` |
| Model type | Continued MLM fine-tune |
| Transformer layers | 36 |
| Hidden size | 1,152 |
| Attention heads | 18 |
| ESMCapsid representation | Layer 35, attention-mask mean pooling |

## Usage

### Installation

```bash
pip install torch transformers tokenizers
```

### Load the model and extract Layer 35 embeddings

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_id = "Shuofang127/ESMCapsid-C"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
).eval()

batch = tokenizer(
    ["MSTNPKPQRKTKRNT", "MKTIIALSYIFCLVFA"],
    padding=True,
    return_tensors="pt",
)

with torch.inference_mode():
    output = model(**batch, output_hidden_states=True)

# The packaged implementation returns block states without an input-embedding state.
layer35 = output.hidden_states[35]
mask = batch["attention_mask"].unsqueeze(-1).to(layer35.dtype)
embedding = (layer35 * mask).sum(dim=1) / mask.sum(dim=1).clamp_min(1)

print(embedding.shape)  # (batch_size, 1152)
```

For offline loading, replace `model_id` with the local model directory and set `local_files_only=True` in both `from_pretrained` calls.

## Property heads

Nine pretrained heads are provided in [`heads/property_heads/`](heads/property_heads/):
`realm`, `kingdom`, `phylum`, `class`, `order`, `family`, `genome_label`,
`host_group`, and `fold_label`. Each includes its checkpoint, class mapping,
and prediction threshold.

Use the [ESMCapsid Inference toolkit](https://github.com/liusihang/ESMCapsid/tree/main/Inference)
with `--property-heads /path/to/heads/property_heads` for `annotate` or
`predict`. See the [head package README](heads/property_heads/README.md) for
a heads-only download example. These heads use a distinct layer-33,
residue-only representation computed internally by the toolkit, not the
public layer-35 embedding vectors. They are auxiliary model assets and do
not change `AutoModel.from_pretrained()`.

## Important

- ESMCapsid-C is a representation model, not the first-stage capsid classifier. Use [`Shuofang127/ESMCapsid-S`](https://huggingface.co/Shuofang127/ESMCapsid-S) for candidate screening.
- Embedding similarity does not establish infectivity, host range, taxonomy, structure, or biological function.
- Keep the released layer, sequence preprocessing, truncation policy, and pooling method together.
- This repository uses custom code and requires `trust_remote_code=True`. Review the code before loading it.
- The packaged `modeling_esm_plusplus.py` is the runtime implementation for this release; features added later to the base repository are not automatically supported here.

## Citation

- Base model: [`Synthyra/ESMplusplus_large`](https://huggingface.co/Synthyra/ESMplusplus_large), DOI [`10.57967/hf/3726`](https://doi.org/10.57967/hf/3726).
- Hallee et al. *FastPLMs: Bringing ESMC and ESM3 to Transformers*. DOI [`10.5281/zenodo.16466575`](https://doi.org/10.5281/zenodo.16466575).
- Liu, S., Xia, S., and Wang, H. *Capsid-specialized protein language models reveal higher-order viral architecture from sequence*. bioRxiv (2026). DOI [`10.64898/2026.09.06.749605`](https://doi.org/10.64898/2026.09.06.749605).

## License

ESMCapsid-C is a derivative of ESM++ Large and ESMC 600M. Review the repository's `LICENSE` and `NOTICE.txt` before use or redistribution.