{"cells":[{"cell_type":"markdown","id":"9417c6c6-ad01-47a7-baa7-f52933e26cc1","metadata":{},"source":"Language Models with n-grams\n============================\n\n**Author:** Joseph Le Roux\n\n**Date:** 2025-11-19\n\n"},{"cell_type":"markdown","id":"a0db2b9d-1224-4246-9688-1880d0c8a66a","metadata":{},"source":["- AUTHOR: Joseph Le Roux\n","- DATE: 2025-11-19\n","- DESCRIPTION: Language Models with n-grams\n"]},{"cell_type":"markdown","id":"25e8ab1f-529e-4883-9948-372168242d6d","metadata":{},"source":"## Table of Contents\n\n- [Introduction](#intro)\n    - [Load Corpus](#chargement-du-corpus)\n    - [*N-gram* dataset](#le-jeu-de-données-n-gramme)\n- [Neural  *n*-gram LM](#modèle-de-langue-neuronal-n-gramme)\n- [Dataset and Training](#création-des-jeux-de-données)\n"},{"cell_type":"markdown","id":"cd4fd34a-0f98-47c7-ba8b-79e4f25045b2","metadata":{},"source":["## Introduction:PROPERTIES:\n\n"]},{"cell_type":"code","execution_count":1,"id":"072d920e-c975-4f90-be00-5c0d74e3f03a","metadata":{},"outputs":[],"source":["!pip uninstall -y torchvision torchaudio numpy torch torchtext\n!pip install torch==2.2.0 torchtext==0.17.0\nimport torch\nimport torchtext\n\nfrom collections import Counter, OrderedDict\nimport itertools"]},{"cell_type":"markdown","id":"99ed8bca-ce1f-4b7b-8336-dc07fa235b97","metadata":{},"source":["The goal of the tutorial is to compare  n-gram models, count-based and neuronal, that you shoud implement first.\n\n"]},{"cell_type":"code","execution_count":1,"id":"fc566032-523f-47d9-8feb-762704e19545","metadata":{},"outputs":[],"source":["N=3 # to compute N-gram models"]},{"cell_type":"markdown","id":"9c977013-b04a-4be6-82e5-dfef819e0518","metadata":{},"source":["### Load Corpus:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"c35a3cb4-44a7-43b2-8f11-ed78602fac99","metadata":{},"source":["Download copyright-free books (in French) from Gutenberg:\n\n"]},{"cell_type":"code","execution_count":1,"id":"37c6143b-1242-4daf-8f2b-9800391893cd","metadata":{},"outputs":[],"source":["!wget https://lipn.fr/~leroux/INFO3_DL4NLP/tps/data/lm_corpus.txt.gz\n!gunzip lm_corpus.txt.gz\n!head -n 20 lm_corpus.txt"]},{"cell_type":"markdown","id":"a51a7f72-0d9b-4fcb-b9cf-4413d35abf70","metadata":{},"source":["This creates lists of sentences for training and evaluation, and a *vocabulary*, conversion tables between strings and identifiers\n(*torchtext.vocab.vocab* class)\n\nWhat are the parameters for?\n\n"]},{"cell_type":"code","execution_count":1,"id":"d7a0d089-5337-4a6c-9fcc-481596cfd7d0","metadata":{},"outputs":[],"source":["def load_corpus(filename, maxsentence=-1, threshold=2, ratio_train=0.9,\n                ngram=3, maxlength=100,\n                unk_string='<unk>', pad_string='<pad>',\n                bos_string='<bos>', eos_string='<eos>'):\n  corpus = []\n  with open(filename, \"r\") as f:\n    sent_counter = 0\n    line = f.readline()\n    while ((maxsentence <0)  or (sent_counter < maxsentence)) and line:\n      sent_counter += 1\n      corpus.append((bos_string + \" \") * (ngram-1) + line.lower() + \" \" + eos_string)\n      line = f.readline()\n\n\n  corpus = list(torchtext.data.functional.simple_space_split(corpus))\n\n  corpus = [x for x in corpus if len(x) < maxlength]\n\n  sep = int(ratio_train * len(corpus))\n  train,dev = (corpus[:sep], corpus[sep:])\n\n  counter = Counter(list(itertools.chain.from_iterable(train)))\n  sorted_by_freq_tuples = sorted(counter.items(), key=lambda x: x[1], reverse=True)\n  sorted_by_freq_tuples = [x for x in sorted_by_freq_tuples if x[1] >  threshold] #threshold]\n  ordered_dict = OrderedDict(sorted_by_freq_tuples)\n  vocabulary = torchtext.vocab.vocab(ordered_dict,specials=[unk_string, pad_string, bos_string, eos_string], special_first=True)\n  vocabulary.set_default_index(vocabulary[unk_string])\n\n  train = [vocabulary.lookup_indices(x) for x in train]\n  dev = [vocabulary.lookup_indices(x) for x in dev]\n\n\n  return train,dev,vocabulary"]},{"cell_type":"code","execution_count":1,"id":"045817dd-75e3-4357-b920-caa4f296f495","metadata":{},"outputs":[],"source":["train, dev,vocabulary = load_corpus(\"lm_corpus.txt\", 100000,2, ngram=N)"]},{"cell_type":"code","execution_count":1,"id":"bf80e54c-ff76-47cc-89c3-461be1e573d3","metadata":{},"outputs":[],"source":["print(train[:3])"]},{"cell_type":"markdown","id":"99315451-e193-4bb6-ae73-b72eafe7e274","metadata":{},"source":["### *N-gram* dataset:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"3b1522e4-fda6-4ae3-b747-a4dcf59d8682","metadata":{},"source":["Split sentences in pairs\n$[(w_i,\\ldots,w_{i+N-2}),w_{i+N-1}]$, (N-1 words, 1 word) stored in a *torch.utils.data.Dataset*\n\n"]},{"cell_type":"code","execution_count":1,"id":"5e1437d8-6de7-49e5-9825-18284d70e0cf","metadata":{},"outputs":[],"source":["class NgramDataset(torch.utils.data.Dataset):\n  def __init__(self, n, sentences):\n    super().__init__()\n    self.examples = []\n    for sentence in sentences:\n      for i in range(0,len(sentence)-(n-1)):\n        self.examples.append((torch.tensor(sentence[i:i+n-1]), sentence[i+n-1]))\n\n  def __len__(self):\n    return len(self.examples)\n\n  def __getitem__(self,k):\n    return self.examples[k]"]},{"cell_type":"markdown","id":"06d44fec-c7aa-4a7e-abfb-bf2d2396cb52","metadata":{},"source":["## Neural  *n*-gram LM:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"1c8bcc61-dffe-48ee-a959-3fa13938793d","metadata":{},"source":["Prediction of the k<sup>th</sup> word is performed by concatenating vectors of the $(n-1)$ previous word\nand using the concatentation as input for classification MLP with $|V|$ classes.\nYou will implement\n\n-   the constructor\n-   the `forward` function\n-   the `generate` method that generates a sentences randomly by sampling each position until `<EOS>` is generated, or the maximum authorized length is reached.\n-   the `compute_perplexity` method computing the `perplexity` of a corpus\n\n"]},{"cell_type":"code","execution_count":1,"id":"c04ae009-1733-492b-9e4c-9c477d6708cd","metadata":{},"outputs":[],"source":["class NeuralNgramLM(torch.nn.Module):\n  def __init__(self, n, vocab_size, word_emb_dim, hidden_dim, dropout_rate, device):\n    super().__init__()\n    \n\n\n    self.device = device\n    self = self.to(self.device)\n\n  def forward(self, x):\n    \n    return x\n\n  def generate(self, maxlength):\n    sentence = []\n    \n    return sentence\n\n  def compute_perplexity(dl):"]},{"cell_type":"markdown","id":"1309de79-b886-49a6-bdd4-45baf4b1bfb7","metadata":{},"source":["## Dataset and Training:PROPERTIES:\n\n"]},{"cell_type":"code","execution_count":1,"id":"c9a4206f-df59-48d8-acaa-a6762c2cb569","metadata":{},"outputs":[],"source":["train_dataset = NgramDataset(N,train)\ndev_dataset = NgramDataset(N,dev)"]},{"cell_type":"markdown","id":"c71ed8e2-9311-4f1c-9189-5ce95668e4ee","metadata":{},"source":["The following code implements the training loop. After each epoch, we compute\nthe perplexity on the development set. The best model to save should be the one\nwith the lowest perplexity. Alos the model generates 5 sentences after each\nepoch to have a feel of the fluency level.\n\n"]},{"cell_type":"code","execution_count":1,"id":"d0893f27-33ec-4935-9d15-74cbb514d72c","metadata":{},"outputs":[],"source":["def train(model, opt, tds,dds, nb_epochs, batch_size, collate_fn=None):\n  train_dl = torch.utils.data.DataLoader(tds, batch_size=batch_size,\n                                         shuffle=True, collate_fn=collate_fn)\n\n  dev_dl = torch.utils.data.DataLoader(dds, batch_size=batch_size*2,\n                                         shuffle=False, collate_fn=collate_fn)\n\n  loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.0,ignore_index=1)\n\n  for epoch in range(nb_epochs):\n    epoch_loss = 0.0\n    n = 0\n    model.train()\n    for batch_input, batch_output in train_dl:\n      batch_input = batch_input.to(model.device)\n      batch_output = batch_output.to(model.device)\n\n      preds = model(batch_input)\n      #cross_entropy wants to work on a vector, a 1D tensor\n      loss = loss_fn(preds.view(-1,preds.size(-1)), batch_output.view(-1))\n\n      epoch_loss += loss.item()\n      n +=1\n\n      opt.zero_grad()\n      loss.backward()\n      opt.step()\n\n    print(epoch, \"epoch loss: \", epoch_loss / (n*batch_size))\n    model.eval()\n\n    perp = model.compute_perplexity(dev_dl)\n    print(\"Perplexity:\", perp)\n\n    for _ in range(5):\n      s = model.generate(25)\n      print(\" \".join(vocabulary.lookup_tokens(s)))"]},{"cell_type":"markdown","id":"32e910db-1a76-46c6-adeb-d86b92e64f3b","metadata":{},"source":["Putting everything together, we have the final experiment for this tutorial:\n\n"]},{"cell_type":"code","execution_count":1,"id":"e9e8af86-51aa-43c8-ab08-7753a009c3e1","metadata":{},"outputs":[],"source":["model = NeuralNgramLM(N,len(vocabulary), 200, 512, 0.1, torch.device('cuda:0'))\nopt = torch.optim.Adam(params=model.parameters(), lr=0.001)\n\nfor _ in range(5):\n  s = model.generate(maxlength=25)\n  print(\" \".join(vocabulary.lookup_tokens(s)))\n\ntrain(model, opt, train_dataset , dev_dataset, 50, 1024)"]}],"metadata":{"org":{"AUTHOR":"Joseph Le Roux","DATE":"2025-11-19","DESCRIPTION":"Language Models with n-grams"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":5}