amin/medical_embedding_1

SentenceTransformer based on abhinand/MedEmbed-small-v0.1

This is a sentence-transformers model finetuned from abhinand/MedEmbed-small-v0.1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: abhinand/MedEmbed-small-v0.1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What are the symptoms of Presti?',
    'the fetal chest, a four-chamber view of the heart is imaged. Note that the apex of the heart is pointing toward the left side of the fetal chest (Figs. 6.2 and 6.4). Determining that the stomach, descending aorta, and cardiac apex are located on the fetal left side and the inferior vena cava is located on the right side establishes normal visceral situs (Figs. 6.1 and 6.3). Figure 6.1: Schematic drawing of a cross section of the upper abdomen for the assessment of the abdominal situs. The vertical line divides this plane into right and left. The right-sided structures include the gallbladder, the portal sinus, a large part of the liver, and inferior vena cava (IVC). The left-sided structures include the descending aorta, the stomach, and the spleen. Figure 6.3 is the corresponding ultrasound plane. Figure 6.2: Determining fetal situs in longitudinal lie: In A, the fetus is in a cephalic presentation with the fetal spine close to the left uterine wall, resulting in the right side being anterior and left side posterior. In B, the fetus is in a cephalic presentation with the fetal spine close to the right uterine wall, resulting in the left side being anterior and right side posterior. In C, the fetus is in a breech presentation with the fetal spine close to the left uterine wall, resulting in the left side being anterior and right side posterior. In D, the fetus is in a breech presentation with the fetal spine close to the right uterine wall, resulting in the right side being anterior and left side posterior. Note the corresponding transverse ultrasound planes of the chest and abdomen. Blue arrows point to fetal stomach, red arrows to the apex of the heart, and yellow arrows to the descending aorta. See text for details. Several',
    'diaphragmatic hernia has been reported.67,77,78 Although these malformations may be detected in the first trimester, visualization will depend on size, and continued growth may aid detection in the second trimester. In a randomized trial of routine 12-week anatomic survey versus routine 18-week anatomic survey, Saltvedt and colleagues detected 0% of the three diaphragmatic hernias in the 12-week group but 50% of four diaphragmatic hernias in the 18-week group, but this difference was not statistically signifi- cant because of the overall low prevalence of congenital diaphragmatic hernia in the cohort (7/36,108).78 Cardiac Disease Congenital heart disease is one of the most common severe congenital abnormalities, with a prevalence of 8/1000 live births.22,72,79,80 Over the past 2 decades, imaging of the fetal heart in the first trimester has evolved considerably to include full echocardiographic studies, with several authors reporting diagnosis of congenital heart disease in the first trimester22,30,79-81 (Fig. 5-23). In a retrospective study of 2165 sin- gleton pregnancies that underwent fetal echocardiogram from 1997 to 2003 Smrcek and colleagues reported the frequency of congenital heart malformations diagnosed between 11 and 13 weeks, with atrioven- tricular septal defects being the most frequent by about 4.5-fold (18/29), followed by ventricular septal defect (4/29), and tetralogy of Fallot (3/29).28 Additionally, ectopia cordis, hypoplastic left-sided and right-sided heart syndrome, double outlet right ventricle, transposi- tion of the great arteries, absence of the pulmonary valves, aortic ste- nosis, aortic coarctation, left and right atrial isomerism, pulmonary stenosis, truncus arteriosus, tricuspid atresia, and total anomalous pul- monary venous return have all been reported as either isolated findings or in combination as complex congenital heart disease.22,28,81-83 The majority of studies evaluating first trimester fetal cardiac evaluation have included a selected population referred for specialized fetal echo- cardiogram in which the indication most commonly was increased nuchal translucency but',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.8767
spearman_cosine	0.6635

Training Details

Training Dataset

Unnamed Dataset

• Size: 16,156 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 6 tokens mean: 10.24 tokens max: 25 tokens	min: 292 tokens mean: 477.35 tokens max: 512 tokens	min: 0.0 mean: 0.15 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  What are the symptoms of Obstet Gynecol?	Imaging Parameters The ACR practice parameters for the performance of ce-MR imaging were revised in 2013 and amended in 2014. Table 22.4 lists the performance guidelines by technical factor. For a facility to be accredited for breast MR, they have to follow the ACR guidelines, but specific protocols will vary across institutions. In addition, for ACR accreditation, they must be able to do mammographic correlation, breast US, and MR imagingguided procedures or have a relationship with a facility that can provide those services for them. MR imaging equipment specifications and performance must also meet all state and federal requirements. Patients are scanned in the prone position with the breasts hanging into a dedicated breast coil. Body coils should not be used for breast MR examinations. The breast should be imaged in axial or sagittal planes or a combination of the two. Core pulse sequences when evaluating the breast for cancer include a three-plane localizer, T1W images, T2W images...	0.0
  What is diagnosis?	(CNS) organs should be per- formed for differential diagnosis among syndromes presenting with fetal skeletal anomalies. For example, congenital heart disease is a prominent feature of Ellisvan Creveld and Holt-Oram syndromes.252,253 Fetal Movements The normal pattern of fetal movements can be identified as early as 11 weeks of gestation through a detailed anatomic evaluation.254-257 Abnormal fetal movements can be observed in skeletal disorders involving joint contractures, neural muscular connective tissue disor- ders, amyoplasia (lack of muscle growth), vascular compromise, and anomalies of the spinal cord. The most frequent conditions associated with abnormal or absent fetal movements are fetal akinesia deforma- tion sequence (FADS) or Pena-Shokeir syndrome, and arthrogrypo- sis.258 In FADS there is a significant reduction in the amplitude, velocity, and complexity of fetal movements.259,260 In arthrogryposis, there is fixed position of the distal parts of the limbs and reduced ampl...	1.0
  What are the risk factors for Diagnostic Ultra?	G, Bast C, Lenz F, Bollmann R. Doppler echocardiography of the main stems of the pulmonary arteries in the normal human fetus. Ultrasound Obstet Gynecol 1998;11: 1739 47. Roth P, Agnani G, Arbez Gindre F, Pauchard JY, Burguet A, Schaal JP, Maillet R. Use of energy color Doppler in visualizing fetal pulmonary vascularization to predict the absence of severe pulmonary hypoplasia Gynecol Obstet Invest 1998;46:1537 48. Chaoui R, Kalache K, Tennstedt C, Lenz F, Vogel M. Pulmonary arterial Doppler in fetuses with lung hypoplasia. Eur J Obstet Gynecol Reprod Biol 1999:84:17985 49. Yoshimura S, Masuzaki H, Miura K, Muta K, Gotoh H, Ishimaru T. Diagnosis of fetal pulmonary hypoplasia by measurement of blood flow velocity waveforms of pulmonary arteries with Doppler ultrasonography. Am J Obstet Gynecol 1999;180:4416 50. Sherer DM, Eglinton GS, Goncalves LF, Lewis KM, Queenan JT. Prenatal color and pulsed Doppler sonographic documentation of intrathoracic umbilical vein and ductus venosus, confir...	0.0

• Loss: ContrastiveLoss with these parameters:

  {
      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
      "margin": 0.5,
      "size_average": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	spearman_cosine
0.4950	500	0.0094	-
0.5	505	-	0.6499
0.9901	1000	0.0052	-
1.0	1010	-	0.6607
1.4851	1500	0.0041	-
1.5	1515	-	0.6597
1.9802	2000	0.0035	-
2.0	2020	-	0.6632
2.4752	2500	0.003	-
2.5	2525	-	0.6631
2.9703	3000	0.0031	-
3.0	3030	-	0.6635

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.6.0+cu124
• Accelerate: 1.7.0
• Datasets: 2.14.4
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveLoss

@inproceedings{hadsell2006dimensionality,
    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)},
    title={Dimensionality Reduction by Learning an Invariant Mapping},
    year={2006},
    volume={2},
    number={},
    pages={1735-1742},
    doi={10.1109/CVPR.2006.100}
}