Hugging Face's logo

rayraycano
/
finetune-demo-lora

Transformers
Safetensors
Generated from Trainer
trl
sft
Model card Files
xet
 Community
finetune-demo-lora
File size: 2,211 Bytes
fcca8c8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
---
title: "Multi-GPU"
format:
  html:
    toc: true
    toc-depth: 3
    number-sections: true
    code-tools: true
execute:
  enabled: false
---

This guide covers advanced training configurations for multi-GPU setups using Axolotl.

## Overview {#sec-overview}

Axolotl supports several methods for multi-GPU training:

- DeepSpeed (recommended)
- FSDP (Fully Sharded Data Parallel)
- FSDP + QLoRA

## DeepSpeed {#sec-deepspeed}

DeepSpeed is the recommended approach for multi-GPU training due to its stability and performance. It provides various optimization levels through ZeRO stages.

### Configuration {#sec-deepspeed-config}

Add to your YAML config:

```{.yaml}
deepspeed: deepspeed_configs/zero1.json
```

### Usage {#sec-deepspeed-usage}

```{.bash}
# Passing arg via config
axolotl train config.yml

# Passing arg via cli
axolotl train config.yml --deepspeed deepspeed_configs/zero1.json
```

### ZeRO Stages {#sec-zero-stages}

We provide default configurations for:

- ZeRO Stage 1 (`zero1.json`)
- ZeRO Stage 2 (`zero2.json`)
- ZeRO Stage 3 (`zero3.json`)

Choose based on your memory requirements and performance needs.

## FSDP {#sec-fsdp}

### Basic FSDP Configuration {#sec-fsdp-config}

```{.yaml}
fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_offload_params: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
```

### FSDP + QLoRA {#sec-fsdp-qlora}

For combining FSDP with QLoRA, see our [dedicated guide](fsdp_qlora.qmd).

## Performance Optimization {#sec-performance}

### Liger Kernel Integration {#sec-liger}

Please see [docs](custom_integrations.qmd#liger) for more info.

## Troubleshooting {#sec-troubleshooting}

### NCCL Issues {#sec-nccl}

For NCCL-related problems, see our [NCCL troubleshooting guide](nccl.qmd).

### Common Problems {#sec-common-problems}

::: {.panel-tabset}

## Memory Issues

- Reduce `micro_batch_size`
- Reduce `eval_batch_size`
- Adjust `gradient_accumulation_steps`
- Consider using a higher ZeRO stage

## Training Instability

- Start with DeepSpeed ZeRO-2
- Monitor loss values
- Check learning rates

:::

For more detailed troubleshooting, see our [debugging guide](debugging.qmd).