hanyeh00 commited on
Commit
a6683ae
·
verified ·
1 Parent(s): 6738482

Upload Hamsan v1.0.0 (v6 merged 0.7 with e5-base)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
1_Pooling/config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "embedding_dimension": 768,
3
+ "pooling_mode": "mean",
4
+ "include_prompt": true
5
+ }
2_Normalize/config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "module_input_name": "sentence_embedding",
3
+ "module_output_name": "sentence_embedding"
4
+ }
README.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - fa
4
+ license: cc-by-sa-4.0
5
+ base_model: intfloat/multilingual-e5-base
6
+ library_name: sentence-transformers
7
+ pipeline_tag: sentence-similarity
8
+ tags:
9
+ - sentence-transformers
10
+ - sentence-similarity
11
+ - feature-extraction
12
+ - persian
13
+ - farsi
14
+ - matryoshka
15
+ widget:
16
+ - source_sentence: 'query: پایتخت ایران کجاست؟'
17
+ sentences:
18
+ - 'passage: تهران پایتخت و پرجمعیت‌ترین شهر ایران است.'
19
+ - 'passage: اصفهان یکی از شهرهای تاریخی ایران است.'
20
+ - 'passage: دریای خزر بزرگ‌ترین دریاچهٔ جهان است.'
21
+ ---
22
+
23
+ # Hamsan (همسان) — Persian sentence embeddings
24
+
25
+ **Hamsan** is a Persian text-embedding model built on
26
+ [`intfloat/multilingual-e5-base`](https://huggingface.co/intfloat/multilingual-e5-base).
27
+ It turns Persian sentences and passages into 768-dimensional vectors for semantic search,
28
+ RAG, clustering, and similarity.
29
+
30
+ همسان یک مدل embedding فارسی است برای جست‌وجوی معنایی، RAG و سنجش شباهت جمله‌ها.
31
+
32
+ - **Same size and speed as e5-base:** 278M parameters, runs on a laptop CPU or a small GPU.
33
+ - **Open license:** CC BY-SA 4.0. No non-commercial training data.
34
+ - **Short vectors:** trained with Matryoshka loss, so vectors can be cut to 512/256/128 dims.
35
+ - **Built-in prompts:** `query` / `passage` prefixes are saved in the model.
36
+
37
+ ## Usage
38
+
39
+ ```python
40
+ from sentence_transformers import SentenceTransformer
41
+
42
+ model = SentenceTransformer("hanyeh00/hamsan")
43
+
44
+ queries = model.encode(["پایتخت ایران کجاست؟"], prompt_name="query")
45
+ docs = model.encode([
46
+ "تهران پایتخت و پرجمعیت‌ترین شهر ایران است.",
47
+ "اصفهان یکی از شهرهای تاریخی ایران است.",
48
+ ], prompt_name="passage")
49
+
50
+ print(model.similarity(queries, docs))
51
+ ```
52
+
53
+ - Questions and search queries: `prompt_name="query"`.
54
+ - Documents and passages to search over: `prompt_name="passage"`.
55
+ - Comparing two sentences with each other: use `"query"` for both.
56
+
57
+ Shorter vectors (smaller index, faster search):
58
+
59
+ ```python
60
+ model = SentenceTransformer("hanyeh00/hamsan", truncate_dim=256)
61
+ ```
62
+
63
+ ## Evaluation
64
+
65
+ Lite [FaMTEB](https://huggingface.co/spaces/mteb/leaderboard) subset (8 retrieval + 2 STS tasks).
66
+ Retrieval: nDCG@10. STS: Spearman.
67
+
68
+ | Task | e5-base | **Hamsan** |
69
+ |---|---|---|
70
+ | SynPerChatbotRAGFAQRetrieval | 0.285 | **0.287** |
71
+ | WikipediaRetrievalMultilingual | **0.881** | 0.861 |
72
+ | MSMARCO-FaHardNegatives | **0.667** | 0.628 |
73
+ | ArguAna-Fa.v2 | 0.338 | **0.494** |
74
+ | FiQA2018-Fa.v2 | **0.230** | 0.227 |
75
+ | SCIDOCS-Fa.v2 | 0.118 | **0.141** |
76
+ | SciFact-Fa.v2 | 0.582 | 0.582 |
77
+ | NeuCLIR2023RetrievalHardNegatives | 0.530 | **0.561** |
78
+ | **Retrieval average** | 0.454 | **0.473** |
79
+ | Farsick (STS) | 0.686 | **0.707** |
80
+ | SynPerSTS (STS) | **0.869** | 0.853 |
81
+ | **STS average** | 0.778 | **0.780** |
82
+
83
+ Hamsan improves the retrieval average over e5-base by about 2 points, with the largest gain on
84
+ argument retrieval (ArguAna, +0.16) and gains on scientific (SCIDOCS) and news (NeuCLIR)
85
+ retrieval. It is slightly weaker on web search (MSMARCO) and Wikipedia retrieval, and on par
86
+ for sentence similarity. Larger models such as BGE-M3 or
87
+ [Hakim](https://huggingface.co/MCINext/Hakim) score higher on FaMTEB.
88
+
89
+ ## Training
90
+
91
+ 1. **Fine-tuning** e5-base for 1 epoch on ~115k Persian pairs and triplets from several domains:
92
+ - Persian Wikipedia (title → paragraph), PQuAD and PersianQA (question → paragraph)
93
+ - [pn_summary](https://huggingface.co/datasets/HooshvareLab/pn_summary) news
94
+ (title / summary → article)
95
+ - [persian-med-qa](https://huggingface.co/datasets/aictsharif/persian-med-qa)
96
+ (medical question → answer)
97
+ - synthetic STS triplets
98
+
99
+ Hard negatives were mined with the base model. Every batch comes from a single source,
100
+ so in-batch negatives are from the same domain.
101
+ Loss: `MatryoshkaLoss(CachedMultipleNegativesRankingLoss)`, batch 128, lr 7e-6,
102
+ max length 256, token embeddings frozen.
103
+ 2. **Merging** with the base model to keep its general knowledge:
104
+ `weights = 0.7 × fine-tuned + 0.3 × e5-base`.
105
+
106
+ ## Limitations
107
+
108
+ - Only Persian was evaluated; other languages may be weaker than in e5-base.
109
+ - Not a source of medical advice, even though medical Q&A was part of the training data.
110
+ - Inputs longer than 256 tokens are truncated.
111
+
112
+ ## Citation
113
+
114
+ If you use this model, please cite the base model:
115
+
116
+ ```bibtex
117
+ @article{wang2024multilingual,
118
+ title={Multilingual E5 Text Embeddings: A Technical Report},
119
+ author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
120
+ journal={arXiv preprint arXiv:2402.05672},
121
+ year={2024}
122
+ }
123
+ ```
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_cross_attention": false,
3
+ "architectures": [
4
+ "XLMRobertaModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": 0,
8
+ "classifier_dropout": null,
9
+ "dtype": "float32",
10
+ "eos_token_id": 2,
11
+ "hidden_act": "gelu",
12
+ "hidden_dropout_prob": 0.1,
13
+ "hidden_size": 768,
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 3072,
16
+ "is_decoder": false,
17
+ "layer_norm_eps": 1e-05,
18
+ "max_position_embeddings": 514,
19
+ "model_type": "xlm-roberta",
20
+ "num_attention_heads": 12,
21
+ "num_hidden_layers": 12,
22
+ "output_past": true,
23
+ "pad_token_id": 1,
24
+ "position_embedding_type": "absolute",
25
+ "tie_word_embeddings": true,
26
+ "transformers_version": "5.17.0",
27
+ "type_vocab_size": 1,
28
+ "use_cache": false,
29
+ "vocab_size": 250002
30
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "pytorch": "2.14.0+cu126",
4
+ "sentence_transformers": "6.1.0",
5
+ "transformers": "5.17.0"
6
+ },
7
+ "default_prompt_name": null,
8
+ "model_type": "SentenceTransformer",
9
+ "prompts": {
10
+ "document": "passage: ",
11
+ "passage": "passage: ",
12
+ "query": "query: "
13
+ },
14
+ "similarity_fn_name": "cosine"
15
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b1c60c3b8f22a3471a73e6109497a749490ffd0152355466ea5cd13e12838cb2
3
+ size 1112197096
modules.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.base.modules.transformer.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
13
+ },
14
+ {
15
+ "idx": 2,
16
+ "name": "2",
17
+ "path": "2_Normalize",
18
+ "type": "sentence_transformers.base.modules.normalize.Normalize"
19
+ }
20
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "transformer_task": "feature-extraction",
3
+ "modality_config": {
4
+ "text": {
5
+ "method": "forward",
6
+ "method_output_name": "last_hidden_state"
7
+ }
8
+ },
9
+ "module_output_name": "token_embeddings"
10
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a514807cffabd8abaf028cfaffe7ff0c4f60b97ea2db80c41f14172ae6b018ca
3
+ size 17082987
tokenizer_config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": true,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<s>",
5
+ "clean_up_tokenization_spaces": true,
6
+ "cls_token": "<s>",
7
+ "eos_token": "</s>",
8
+ "is_local": true,
9
+ "local_files_only": true,
10
+ "mask_token": "<mask>",
11
+ "max_length": 256,
12
+ "model_max_length": 256,
13
+ "pad_to_multiple_of": null,
14
+ "pad_token": "<pad>",
15
+ "pad_token_type_id": 0,
16
+ "padding_side": "right",
17
+ "sep_token": "</s>",
18
+ "stride": 0,
19
+ "tokenizer_class": "XLMRobertaTokenizer",
20
+ "truncation_side": "right",
21
+ "truncation_strategy": "longest_first",
22
+ "unk_token": "<unk>"
23
+ }