Hugging Face's logo

ulkaa
/
gemma-2b-dolly-qa

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files
xet
 Metrics Community
unrahul commited on
Commit
824a4ce
·
verified ·
1 Parent(s): 2d7c00a

Model save

Browse files
Files changed (4) hide show
  1. README.md +3 -27
  2. adapter_config.json +4 -4
  3. adapter_model.safetensors +1 -1
  4. training_args.bin +1 -1
README.md CHANGED
@@ -6,8 +6,6 @@ tags:
6
  - sft
7
  - generated_from_trainer
8
  base_model: google/gemma-2b
9
- datasets:
10
- - generator
11
  model-index:
12
  - name: gemma-2b-dolly-qa
13
  results: []
@@ -18,9 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
18
 
19
  # gemma-2b-dolly-qa
20
 
21
- This model is a fine-tuned version of [google/gemma-2b](https://huggingface.co/google/gemma-2b) on the generator dataset.
22
- It achieves the following results on the evaluation set:
23
- - Loss: 5.8003
24
 
25
  ## Model description
26
 
@@ -39,7 +35,7 @@ More information needed
39
  ### Training hyperparameters
40
 
41
  The following hyperparameters were used during training:
42
- - learning_rate: 0.0002
43
  - train_batch_size: 2
44
  - eval_batch_size: 8
45
  - seed: 42
@@ -48,27 +44,7 @@ The following hyperparameters were used during training:
48
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
49
  - lr_scheduler_type: linear
50
  - lr_scheduler_warmup_ratio: 0.05
51
- - training_steps: 1480
52
-
53
- ### Training results
54
-
55
- | Training Loss | Epoch | Step | Validation Loss |
56
- |:-------------:|:-----:|:----:|:---------------:|
57
- | 2.3815 | 1.64 | 100 | 2.0042 |
58
- | 1.934 | 3.28 | 200 | 2.0421 |
59
- | 1.7512 | 4.92 | 300 | 2.1031 |
60
- | 1.5047 | 6.56 | 400 | 2.3441 |
61
- | 1.276 | 8.2 | 500 | 2.6745 |
62
- | 1.0163 | 9.84 | 600 | 2.9221 |
63
- | 0.7745 | 11.48 | 700 | 3.3954 |
64
- | 0.5846 | 13.11 | 800 | 4.0620 |
65
- | 0.4107 | 14.75 | 900 | 4.2286 |
66
- | 0.2873 | 16.39 | 1000 | 4.6660 |
67
- | 0.2062 | 18.03 | 1100 | 5.0101 |
68
- | 0.1397 | 19.67 | 1200 | 5.3291 |
69
- | 0.1046 | 21.31 | 1300 | 5.6700 |
70
- | 0.0788 | 22.95 | 1400 | 5.8003 |
71
-
72
 
73
  ### Framework versions
74
 
 
6
  - sft
7
  - generated_from_trainer
8
  base_model: google/gemma-2b
 
 
9
  model-index:
10
  - name: gemma-2b-dolly-qa
11
  results: []
 
16
 
17
  # gemma-2b-dolly-qa
18
 
19
+ This model is a fine-tuned version of [google/gemma-2b](https://huggingface.co/google/gemma-2b) on an unknown dataset.
 
 
20
 
21
  ## Model description
22
 
 
35
  ### Training hyperparameters
36
 
37
  The following hyperparameters were used during training:
38
+ - learning_rate: 1e-05
39
  - train_batch_size: 2
40
  - eval_batch_size: 8
41
  - seed: 42
 
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: linear
46
  - lr_scheduler_warmup_ratio: 0.05
47
+ - training_steps: 296
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ### Framework versions
50
 
adapter_config.json CHANGED
@@ -10,7 +10,7 @@
10
  "layers_to_transform": null,
11
  "loftq_config": {},
12
  "lora_alpha": 16,
13
- "lora_dropout": 0.1,
14
  "megatron_config": null,
15
  "megatron_core": "megatron.core",
16
  "modules_to_save": null,
@@ -19,13 +19,13 @@
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
- "q_proj",
23
- "o_proj",
24
  "k_proj",
25
  "up_proj",
 
26
  "down_proj",
 
27
  "v_proj",
28
- "gate_proj"
29
  ],
30
  "task_type": "CAUSAL_LM",
31
  "use_dora": false,
 
10
  "layers_to_transform": null,
11
  "loftq_config": {},
12
  "lora_alpha": 16,
13
+ "lora_dropout": 0.2,
14
  "megatron_config": null,
15
  "megatron_core": "megatron.core",
16
  "modules_to_save": null,
 
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
 
 
22
  "k_proj",
23
  "up_proj",
24
+ "q_proj",
25
  "down_proj",
26
+ "gate_proj",
27
  "v_proj",
28
+ "o_proj"
29
  ],
30
  "task_type": "CAUSAL_LM",
31
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:90f062828cca54674d96e99dd1357c15ec5c813417795f8c2458469bb634e841
3
  size 156926880
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:29b049eaf43b59ca16cdcca0a9d2d87acb77cb35d19e873e472d66272c282c79
3
  size 156926880
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1e617e264053a3d7ade764cb9a1c722cd3ae128e5f08a31c58b1a091f14b0f2f
3
  size 4856
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98fa1f114646a4077cf36abc71f213ba77366cd3fffe06fc111a9781268be21f
3
  size 4856