kimps005/sms-category-model

CatchSmishing Korean SMS Category Classifier — INT8 Dynamic Quantization

CatchSmishing Korean SMS Category Classifier is a Korean-first, seven-class SMS text classifier for routing messages into common message categories used by the CatchSmishing service. This repository distributes the CPU-oriented INT8 dynamically quantized inference checkpoint.

Important: This is a message-category classifier, not a standalone malicious-message verdict. A PERSONAL prediction does not mean a message is safe, and a finance-, delivery-, or government-themed message is not necessarily fraudulent. Use it with URL, sender, and contextual checks—and a human review process—when the outcome affects a user’s security or finances.

Property	Value
Base model	klue/bert-base
Task	Korean SMS text classification
Labels	7
Input	Korean message text, up to 128 tokens
Deployment artifact	PyTorch INT8 dynamic-quantization state_dict
Target runtime	CPU / PyTorch (tested with PyTorch 2.5.1+cpu)
Checkpoint file	sms_category_model_int8_dynamic_state_dict.pt
Artifact size	178.56 MiB

What this model does

The model assigns one category to a Korean SMS message:

ID	Label	Scope
0	PERSONAL	Everyday, family, friend, and personal messages
1	FINANCE	Banking, card, payment, loan, and other finance-related messages
2	DELIVERY	Shipping, delivery, pickup, return, and travel-delivery notices
3	GOVERNMENT	Government, public-institution, civic, and emergency notices
4	PROMOTION	Advertising, coupons, events, and marketing messages
5	AUTH	Login, account, OTP, password, and security-authentication messages
6	WORK	Workplace, organization, HR, and business-operation messages

It is designed as one signal in the CatchSmishing service, where it is combined with OCR, URL checks, and other risk-analysis signals. It does not perform OCR, QR decoding, URL reputation analysis, sender authentication, or final smishing-risk scoring by itself.

Architecture and quantization

The classifier uses the klue/bert-base configuration (12 transformer layers, 768 hidden size, 12 attention heads, 32,000-token vocabulary) with a custom classification head:

1. [CLS] pooling and attention-mask-aware mean pooling are concatenated.
2. A linear projection with GELU activation produces the representation.
3. Five dropout branches share one 7-class linear classifier; their logits are averaged at inference.

The published checkpoint is a PyTorch state_dict, not a standard AutoModelForSequenceClassification checkpoint. For CPU inference, every torch.nn.Linear module is dynamically quantized to signed INT8 (75 modules in the benchmark); other operations remain in floating point. Therefore, do not try to load it with AutoModelForSequenceClassification.from_pretrained().

Quick start

Install the tested runtime dependencies:

pip install "torch==2.5.1" "transformers==5.3.0" "huggingface_hub==1.7.2"

The following example downloads the files at the release revision, rebuilds the custom model, applies the same INT8 dynamic quantization, and runs one prediction. It intentionally uses weights_only=True when reading the checkpoint.

import re
from pathlib import Path

import torch
import torch.nn as nn
from huggingface_hub import snapshot_download
from transformers import AutoConfig, AutoModel, AutoTokenizer

REPO_ID = "kimps005/sms-category-model"
REVISION = "b0267b3befd229165127de5cf402414d427863fa"
CHECKPOINT = "sms_category_model_int8_dynamic_state_dict.pt"
MAX_LENGTH = 128
LABELS = [
    "PERSONAL", "FINANCE", "DELIVERY", "GOVERNMENT",
    "PROMOTION", "AUTH", "WORK",
]


class SMSClassifier(nn.Module):
    def __init__(self, config, num_dropouts=5):
        super().__init__()
        self.encoder = AutoModel.from_config(config)
        hidden_size = self.encoder.config.hidden_size
        self.dropouts = nn.ModuleList(
            [nn.Dropout(0.3 + index * 0.04) for index in range(num_dropouts)]
        )
        self.proj = nn.Sequential(nn.Linear(hidden_size * 2, hidden_size), nn.GELU())
        self.classifier = nn.Linear(hidden_size, len(LABELS))

    def forward(self, input_ids, attention_mask):
        hidden_states = self.encoder(
            input_ids=input_ids, attention_mask=attention_mask
        ).last_hidden_state
        cls_pool = hidden_states[:, 0, :]
        expanded_mask = attention_mask.unsqueeze(-1).expand(hidden_states.size()).float()
        mean_pool = (hidden_states * expanded_mask).sum(1) / expanded_mask.sum(1).clamp_min(1e-9)
        representation = self.proj(torch.cat([cls_pool, mean_pool], dim=-1))
        return sum(self.classifier(dropout(representation)) for dropout in self.dropouts) / len(self.dropouts)


model_dir = Path(snapshot_download(repo_id=REPO_ID, revision=REVISION))
tokenizer = AutoTokenizer.from_pretrained(model_dir, local_files_only=True)
config = AutoConfig.from_pretrained(model_dir, local_files_only=True)
model = SMSClassifier(config).cpu()

if "x86" in torch.backends.quantized.supported_engines:
    torch.backends.quantized.engine = "x86"
elif "fbgemm" in torch.backends.quantized.supported_engines:
    torch.backends.quantized.engine = "fbgemm"

model = torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)
state_dict = torch.load(model_dir / CHECKPOINT, map_location="cpu", weights_only=True)
model.load_state_dict(state_dict, strict=True)
model.eval()

text = "[국세청] 환급금 지급 예정 안내입니다. 계좌 등록 여부를 확인해 주세요."
text = re.sub(r"https?://\S+", "", text)  # service-aligned preprocessing
encoded = tokenizer(text, max_length=MAX_LENGTH, padding="max_length", truncation=True, return_tensors="pt")

with torch.inference_mode():
    probabilities = model(**encoded).softmax(dim=-1)[0]

label_id = int(probabilities.argmax())
print({"label": LABELS[label_id], "confidence": float(probabilities[label_id])})

For the maintained service integration—including its optional keyword calibration—see server/predictor.py. The service removes URLs and normalizes whitespace before tokenization, uses max_length=128, and runs on CPU.

Performance and evaluation

Quantization trade-off

The local quantization benchmark compares the original FP32 checkpoint with this INT8 dynamic checkpoint. Artifact-size reduction is substantial, while the observed category-classification change is small on this diagnostic benchmark.

Measure	FP32	INT8 dynamic	Change
Checkpoint size	426.58 MiB	178.56 MiB	−58.14%
Accuracy	98.68%	98.57%	−0.11 pp
Macro F1	98.68%	98.57%	−0.11 pp
Batch throughput	3.02 samples/s	4.34 samples/s	+43.7%
Single-message latency, p50	370.97 ms	196.40 ms	−47.1%
Single-message latency, p95	449.58 ms	220.32 ms	−51.0%
Process RSS after model is ready	1,144.58 MiB	923.78 MiB	−19.3%

Benchmark protocol

• Data: 2,800 generated Korean SMS texts, balanced at 400 samples for each of the seven categories. The generator uses template-based texts and reserved .test URLs; it is not a real-world or independently curated test set.
• Pipeline: URL removal, whitespace normalization, tokenizer truncation/padding to 128 tokens, and the service’s class-specific keyword calibration were applied to both variants.
• Performance measurement: batch size 32; 140 single-message latency measurements (20 per class) after 10 warm-up predictions.
• Environment: Windows 10, Python 3.11.9, PyTorch 2.5.1+cpu, x86 quantized engine, one PyTorch thread.
• Reproduction: benchmark_quantization.py and its local result files were used. Run FP32 and INT8 variants in separate processes, as the script requires, to keep RSS values independent.

These measurements establish the expected FP32-to-INT8 trade-off only for the stated environment and generated benchmark. They must not be interpreted as an independently validated real-world smishing-detection rate, an OCR evaluation, or a safety guarantee.

Training information

The reference training workflow is documented in smishing_colab.ipynb. Its configured training recipe is:

Setting	Value
Base model	klue/bert-base
Maximum sequence length	128
Train/test split setting	80% / 20%
Epoch limit	12, with early stopping (patience 4)
Batch / gradient accumulation	32 / 2 (effective batch size 64)
Learning rate	2e-5
Dropout	0.30 to 0.46 across five branches
Label smoothing	0.10
Focal-loss gamma	1.5
Seed	42

The notebook takes an externally supplied metadata.csv and adds hand-authored category examples. The exact training CSV, raw-message provenance, split membership, and training log for this released artifact are not currently published. This is intentionally stated here rather than inferred: users should treat the model as a project artifact, not as a fully data-documented benchmark model.

Intended use

Appropriate uses include:

• Routing Korean SMS messages to downstream category-specific checks.
• Helping analysts triage messages alongside URL/domain, sender, and user-context signals.
• CPU-constrained prototype or service deployments where a smaller checkpoint is valuable.

Do not use this model as the sole basis for:

• Blocking messages, closing accounts, or taking punitive action.
• Confirming that a message, sender, URL, or institution is legitimate.
• Making legal, financial, or emergency decisions without verification through an official channel.

Limitations, risks, and responsible use

• Category ≠ threat status. Legitimate and fraudulent messages can share the same category. In particular, personal/relationship impersonation may be labelled PERSONAL.
• Language and domain coverage. The model is Korean-first and is not validated for other languages, code-switched text, obfuscated spellings, unusual Unicode, or new social-engineering campaigns.
• Synthetic evaluation bias. The reported benchmark is templated and may share vocabulary or structure with development data. Its scores can overstate performance on live SMS traffic.
• Rule calibration changes outputs. The service adds category-specific keyword boosts after the neural model. The published benchmark includes those boosts; a bare checkpoint integration can produce different probabilities and labels.
• URLs are removed for classification. This artifact does not inspect a URL’s destination or reputation. Analyze links separately and do not treat a category result as a URL-safety assessment.
• Confidence is not calibrated risk. Softmax confidence should not be interpreted as the probability that a message is fraudulent or safe.
• Privacy. SMS content can include phone numbers, names, financial information, and one-time codes. Minimize retention, secure logs, and obtain the appropriate consent before sending data to any remote service.

Files expected in this repository

File	Purpose
sms_category_model_int8_dynamic_state_dict.pt	INT8 dynamic-quantized model weights
config.json	klue/bert-base architecture configuration
tokenizer.json, tokenizer_config.json, vocab.txt, special_tokens_map.json	Tokenizer assets
README.md	This model card

License and citation

No license file or SPDX license declaration is currently included with this model repository. Do not assume redistribution or commercial-use rights; contact the repository owner before reuse beyond the intended project context. The base model may have separate terms that also apply.

There is no associated paper. If this artifact is useful in a project or report, link to the model repository and the CatchSmishing source repository.