🤗 Speed up
Having began in a time when wrappers have been much less frequent, I turned accustomed to writing my very own coaching loops, which I discover simpler to debug – an method that 🤗 Speed up helps successfully. It proved useful on this mission – I wasn’t fully sure of the required information and label codecs or shapes and my information didn’t match the well-organized examples typically proven in tutorials, however having full entry to intermediate computations through the coaching loop allowed me to iterate shortly.
Context Size
Most tutorials recommend utilizing every sentence as a single coaching instance. Nonetheless, on this case, I made a decision an extended context could be extra appropriate as paperwork usually comprise references to a number of entities, a lot of that are irrelevant (e.g. legal professionals, different collectors, case numbers). This broader context allows the mannequin to raised establish the related shopper. I used 512 tokens from every doc as one coaching instance. This can be a frequent most restrict for fashions however comfortably accommodates all entities in most of my paperwork.
Labelling of Subtokens
Within the 🤗 token classification tutorial [1], advisable method is:
Solely labeling the primary token of a given phrase. Assign
-100to different subtokens from the identical phrase.
Nonetheless, I discovered that the next technique advised within the token classification tutorial of their NLP course [2] works a lot better:
Every token will get the identical label because the token that began the phrase it’s inside, since they’re a part of the identical entity. For tokens inside a phrase however not initially, we change the
B-withI-
Label “-100” is particular label that’s ignored by loss operate. Therefore, I used their features with minor adjustments:
def align_labels_with_tokens(labels, word_ids):
new_labels = []
current_word = None
for word_id in word_ids:
if word_id != current_word:
# Begin of a brand new phrase!
current_word = word_id
label = -100 if word_id is None else labels[word_id]
new_labels.append(label)
elif word_id is None:
# Particular token
new_labels.append(-100)
else:
# Identical phrase as earlier token
label = labels[word_id]
# If the label is B-XXX we modify it to I-XXX
if label % 2 == 1:
label += 1
new_labels.append(label)return new_labels
def tokenize_and_align_labels(examples):
tokenizer = AutoTokenizer.from_pretrained("../mannequin/xlm-roberta-large")
tokenized_inputs = tokenizer(
examples["tokens"], truncation=True, is_split_into_words=True,
padding="max_length", max_length=512)
all_labels = examples["ner_tags"]
new_labels = []
for i, labels in enumerate(all_labels):
word_ids = tokenized_inputs.word_ids(i)
new_labels.append(align_labels_with_tokens(labels, word_ids))
tokenized_inputs["labels"] = new_labels
return tokenized_inputs
I additionally used their postprocess()operate:
To simplify its analysis half, we outline this
postprocess()operate that takes predictions and labels and converts them to lists of strings.
def postprocess(predictions, labels):
predictions = predictions.detach().cpu().clone().numpy()
labels = labels.detach().cpu().clone().numpy()true_labels = [[id2label[l] for l in label if l != -100] for label in labels]
true_predictions = [
[id2label[p] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
return true_predictions, true_labels
Class Weights
Incorporating class weights into the loss operate considerably improved mannequin efficiency. Whereas this adjustment could appear easy — with out it, the mannequin overemphasized the bulk “O” class — it’s surprisingly absent from most tutorials. I applied a customized compute_weights() operate to deal with this imbalance:
def compute_weights(trainset, num_labels):
c = Counter()
for t in trainset:
c += Counter(t['labels'].tolist())
weights = [sum(c.values())/(c[i]+1) for i in vary(num_labels)]
return weights
Coaching Loop
I outlined two extra features: PyTorch DataLoader() to handle batch processing, and a important() operate to arrange distributed coaching objects and execute the coaching loop.
from speed up import Accelerator, notebook_launcher
from collections import Counter
from datasets import Dataset
from datetime import datetime
import torch
from torch.optim.lr_scheduler import ReduceLROnPlateau
from torch.nn import CrossEntropyLoss
from torch.utils.information import DataLoader
from transformers import AutoTokenizer
from transformers import AutoModelForTokenClassification
from transformers import XLMRobertaConfig, XLMRobertaForTokenClassification
from seqeval.metrics import classification_report, f1_scoredef create_dataloaders(trainset, evalset, batch_size, num_workers):
train_dataloader = DataLoader(trainset, shuffle=True,
batch_size=batch_size, num_workers=num_workers)
eval_dataloader = DataLoader(evalset, shuffle=False,
batch_size=batch_size, num_workers=num_workers)
return train_dataloader, eval_dataloader
def important(batch_size, num_workers, epochs, model_path, dataset_tr, dataset_ev, training_type, model_params, dt):
accelerator = Accelerator(split_batches=True)
num_labels = model_params['num_labels']
# Put together information #
train_ds = Dataset.from_dict(
{"tokens": [d[2][:512] for d in dataset_tr],
"ner_tags": [d[1][:512] for d in dataset_tr]})
eval_ds = Dataset.from_dict(
{"tokens": [d[2][:512] for d in dataset_ev],
"ner_tags": [d[1][:512] for d in dataset_ev]})
trainset = train_ds.map(tokenize_and_align_labels, batched=True,
remove_columns=["tokens", "ner_tags"])
evalset = eval_ds.map(tokenize_and_align_labels, batched=True,
remove_columns=["tokens", "ner_tags"])
trainset.set_format("torch")
evalset.set_format("torch")
train_dataloader, eval_dataloader = create_dataloaders(trainset, evalset,
batch_size, num_workers)
# Sort of coaching #
if training_type=='from_scratch':
config = XLMRobertaConfig.from_pretrained(model_path, **model_params)
mannequin = XLMRobertaForTokenClassification(config)
elif training_type=='transfer_learning':
mannequin = AutoModelForTokenClassification.from_pretrained(model_path,
ignore_mismatched_sizes=True, **model_params)
for param in mannequin.parameters():
param.requires_grad=False
for param in mannequin.classifier.parameters():
param.requires_grad=True
elif training_type=='fine_tuning':
mannequin = AutoModelForTokenClassification.from_pretrained(model_path,
**model_params)
for param in mannequin.parameters():
param.requires_grad=True
for param in mannequin.classifier.parameters():
param.requires_grad=True
# Intantiate the optimizer #
optimizer = torch.optim.AdamW(params=mannequin.parameters(), lr=2e-5)
# Instantiate the educational charge scheduler #
lr_scheduler = ReduceLROnPlateau(optimizer, endurance=5)
# Outline loss operate #
weights = compute_weights(trainset, num_labels)
loss_fct = CrossEntropyLoss(weight=torch.tensor(weights))
# Put together objects for distributed coaching #
loss_fct, train_dataloader, mannequin, optimizer, eval_dataloader, lr_scheduler = accelerator.put together(
loss_fct, train_dataloader, mannequin, optimizer, eval_dataloader, lr_scheduler)
# Coaching loop #
max_f1 = 0 # for early stopping
for t in vary(epochs):
# coaching
accelerator.print(f"nnEpoch {t+1}n-------------------------------")
mannequin.prepare()
tr_loss = 0
preds = checklist()
labs = checklist()
for batch in train_dataloader:
outputs = mannequin(input_ids=batch['input_ids'],
attention_mask=batch['attention_mask'])
labels = batch["labels"]
loss = loss_fct(outputs.logits.view(-1, num_labels), labels.view(-1))
accelerator.backward(loss)
optimizer.step()
optimizer.zero_grad()
tr_loss += loss
predictions = outputs.logits.argmax(dim=-1)
predictions_gathered = accelerator.collect(predictions)
labels_gathered = accelerator.collect(labels)
true_predictions, true_labels = postprocess(predictions_gathered, labels_gathered)
preds.prolong(true_predictions)
labs.prolong(true_labels)
lr_scheduler.step(tr_loss)
accelerator.print(f"Prepare loss: {tr_loss/len(train_dataloader):>8f} n")
accelerator.print(classification_report(labs, preds))
# analysis
mannequin.eval()
ev_loss = 0
preds = checklist()
labs = checklist()
for batch in eval_dataloader:
with torch.no_grad():
outputs = mannequin(input_ids=batch['input_ids'],
attention_mask=batch['attention_mask'])
labels = batch["labels"]
loss = loss_fct(outputs.logits.view(-1, num_labels), labels.view(-1))
ev_loss += loss
predictions = outputs.logits.argmax(dim=-1)
predictions_gathered = accelerator.collect(predictions)
labels_gathered = accelerator.collect(labels)
true_predictions, true_labels = postprocess(predictions_gathered, labels_gathered)
preds.prolong(true_predictions)
labs.prolong(true_labels)
accelerator.print(f"Eval loss: {ev_loss/len(eval_dataloader):>8f} n")
accelerator.print(classification_report(labs, preds))
accelerator.print(f"Present Studying Price: {optimizer.param_groups[0]['lr']}")
# checkpoint greatest mannequin
if f1_score(labs, preds) > max_f1:
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(mannequin)
unwrapped_model.save_pretrained(f"../mannequin/xlml_ner/{dt}/",
is_main_process=accelerator.is_main_process,
save_function=accelerator.save)
accelerator.print(f"Mannequin saved throughout {t+1}. epoch.")
max_f1 = f1_score(labs, preds)
best_epoch = t
# early stopping
if (t - best_epoch) > 10:
accelerator.print(f"Early stopping after {t+1}. epoch.")
break
accelerator.print("Finished!")
With all the pieces ready, the mannequin is prepared for coaching. I simply have to provoke the method:
label_list = [
"O",
"B-evcu", "I-evcu", # variable symbol of creditor
"B-rc", "I-rc", # birth ID
"B-prijmeni", "I-prijmeni", # surname
"B-jmeno", "I-jmeno", # given name
"B-datum", "I-datum", # birth date
]
id2label = {a: b for a,b in enumerate(label_list)}
label2id = {b: a for a,b in enumerate(label_list)}num_workers = 6 # variety of GPUs
batch_size = num_workers*2
epochs = 100
model_path = "../mannequin/xlm-roberta-large"
training_type = "fine_tuning" # from_scratch / transfer_learning / fine_tuning
model_params = {"id2label": id2label, "label2id": label2id, "num_labels": 11}
dt = datetime.now().strftime("%Ypercentmpercentd_percentHpercentMpercentS")
os.mkdir(f"../mannequin/xlml_ner/{dt}")
notebook_launcher(important, args=(batch_size, num_workers, epochs, model_path,
dataset_tr, dataset_ev, training_type, model_params, dt),
num_processes=num_workers, mixed_precision="fp16", use_port="29502")
I discover utilizing notebook_launcher() handy, because it permits me to run coaching within the console and simply work with outcomes afterward.
XLM-RoBERTa base vs giant vs Small-E-Czech
I experimented with fine-tuning three fashions. The XLM-RoBERTa base mannequin [3] delivered passable efficiency, however the server capability additionally allowed me to attempt the XLM-RoBERTa giant mannequin [3], which has twice the parameters.
XLM-RoBERTa is a multilingual model of RoBERTa. It’s pre-trained on 2.5TB of filtered CommonCrawl information containing 100 languages.
The big mannequin confirmed a slight enchancment in outcomes, so I finally deployed it. I additionally examined Small-E-Czech [4], an Electra-small mannequin pre-trained on Czech net information, however its efficiency was poor.
High-quality-tuning vs Switch studying vs Coaching from scratch
Along with fine-tuning (updating all mannequin weights), I examined switch studying, as it’s generally advised that coaching solely the ultimate (classification) layer could suffice.. Nonetheless, the efficiency distinction was vital, favoring full fine-tuning. I additionally tried coaching from scratch by importing solely structure of the mannequin, initializing the weights randomly, after which coaching, however as anticipated, this method was ineffective.
RoBERTa vs LLM (Claude 3.5 Sonnet)
I briefly explored zero-shot LLMs, although with minimal immediate engineering (so 🥱). The mannequin struggled even with fundamental requests, similar to (I used Czech within the precise immediate):
Discover variable image of creditor. This quantity has precisely 9 consecutive digits 0–9 with out letters or different particular characters. It’s often preceded by one of many following abbreviations: ‘ev.č.’, ‘zn. opr’, ‘VS. O’, ‘evid. č. opr.’. Quite the opposite, I’m not interested by a transaction quantity with the abbreviation ‘č.j.’. This quantity doesn’t seem typically in paperwork, it could occur that you simply won’t be able to search out it, then write ‘can not discover’. Should you’re undecided, write ‘undecided’.
The mannequin generally did not output the 9-digit format precisely. Submit-processing would filter out shorter numbers, however there have been many false positives 9-digit numbers.
Often the mannequin inferred incorrect delivery IDs based mostly solely on delivery dates (even with temperature set to 0). However, it excelled at extracting names, surnames, and delivery dates.
General, even in my earlier experiments, I discovered that LLMs (on the time of writing) carry out higher with common duties however lack accuracy and reliability for particular or unconventional duties. The efficiency in figuring out the shopper was pretty related for each approaches. For inner causes, the RoBERTa mannequin was deployed.