Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -119,6 +119,11 @@ This port uses [BERT-updated](https://huggingface.co/Taykhoom/BERT-updated), whi
|
|
| 119 |
adds `attn_implementation="sdpa"` and `attn_implementation="flash_attention_2"` support
|
| 120 |
not present in the original codebase.
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
```python
|
| 123 |
model = AutoModel.from_pretrained("Taykhoom/SpliceBERT-1024nt",
|
| 124 |
trust_remote_code=True,
|
|
|
|
| 119 |
adds `attn_implementation="sdpa"` and `attn_implementation="flash_attention_2"` support
|
| 120 |
not present in the original codebase.
|
| 121 |
|
| 122 |
+
The pooler weights (`pooler.dense`) are not present in the original checkpoint and are
|
| 123 |
+
not included in the saved `model.safetensors`. `add_pooling_layer=True` (the default)
|
| 124 |
+
allocates the pooler layer but its weights are randomly initialized -- do not use
|
| 125 |
+
`pooler_output` without fine-tuning.
|
| 126 |
+
|
| 127 |
```python
|
| 128 |
model = AutoModel.from_pretrained("Taykhoom/SpliceBERT-1024nt",
|
| 129 |
trust_remote_code=True,
|