{"cells":[{"cell_type":"markdown","id":"b16ef934-2d99-4d92-b27c-867f0c3411b4","metadata":{},"source":"TP2 Implement Word2vec\n======================\n\n**Author:** Joseph Le Roux\n\n**Date:** 2025-11-19\n\n"},{"cell_type":"markdown","id":"1d120e39-a6ad-4050-aa77-80f2ccc76391","metadata":{},"source":["- AUTHOR: Joseph Le Roux\n","- DATE: 2025-11-19\n","- DESCRIPTION: Implementation of the skip-gram model (MLE)\n"]},{"cell_type":"markdown","id":"4f9571a2-3544-4d59-95f4-5ea263a722f4","metadata":{},"source":"## Table of Contents\n\n- [Fetching the Data](#Fetching-the-Data)\n- [Cleaning and conversion](#Cleaning-and-conversion)\n- [Dataset](#Dataset)\n- [Evaluation](#Evaluation)\n- [The training loop](#The-training-loop)\n- [Creation and training](#Creation-and-training)\n"},{"cell_type":"markdown","id":"c012f7e1-2218-4693-b7b6-537bca899d45","metadata":{},"source":["## Fetching the Data\n\n"]},{"cell_type":"markdown","id":"88331003-2f96-4863-aa17-598f89cd946d","metadata":{},"source":["The following code downloads the data. The commented section retrieves the second corpus, which is unnecessary at first for developing your prototype.\n\n"]},{"cell_type":"code","execution_count":1,"id":"cb977ff5-8ec2-4c89-88d2-9615461f31aa","metadata":{},"outputs":[],"source":["!wget http://mattmahoney.net/dc/text8.zip\n!unzip text8\n!ls"]},{"cell_type":"markdown","id":"6d25ee03-e8f8-4466-a311-d51cc2a02b16","metadata":{},"source":["We display a few lines of the corpus:\n\n"]},{"cell_type":"code","execution_count":1,"id":"b266b4ea-6a63-4b79-b15d-1d0a63776314","metadata":{},"outputs":[],"source":["!wc text8\n!head -c 200 text8"]},{"cell_type":"markdown","id":"cc3c923d-39a1-4215-9010-6438b4122bdb","metadata":{},"source":["## Cleaning and conversion\n\n"]},{"cell_type":"markdown","id":"b86f858d-0119-436c-bf65-f0fc75d3276a","metadata":{},"source":["The following function performs the following tasks:\n\n1.  creation of a wordcount table associating the number of occurrences to each word\n2.  setting a limit on the vocabulary size; the least frequent words are eliminated\n3.  creation of a word2idx table from words to unique integer identifiers\n4.  creation of a symmetric idx2word table\n5.  Finally creates the example corpus in the form of an array of indices\n\n"]},{"cell_type":"code","execution_count":1,"id":"4239578e-b691-49f4-87fe-dd982acb8c2d","metadata":{},"outputs":[],"source":["import random\nimport math\nimport gc\n\nimport torch\n\ndef corpus2train(path, voc_max_size=16384):\n\n  wordcount = {}\n  lines = []\n\n  print(\"counting words\")\n  f = open(path, 'r')\n  for line in f:\n    #words of sentence\n    tokens = line.lower().split()\n    # update counts\n    for t in tokens:\n      if t in wordcount:\n        wordcount[t] +=1\n      else:\n        wordcount[t] = 1\n    lines.append(tokens)\n  f.close()\n\n\n  print(\"remove rare words\")\n  # your code here\n  \n\n  print(\"building tables\")\n  # your code here\n  \n\n  print(\"put all data in a single array\")\n  corpus = sum(lines, [])\n  train = torch.zeros(len(corpus), dtype=torch.int32)\n  for i in range(len(corpus)):\n    train[i] = word2idx[corpus[i]] if corpus[i] in word2idx else voc_max_size\n\n  train = train[train < voc_max_size]\n\n  return(train, word2idx, idx2word, counts)"]},{"cell_type":"code","execution_count":1,"id":"3004ca64-b80b-4480-935e-32f61368fd27","metadata":{},"outputs":[],"source":["trainset,word2idx,idx2word,wordcount = corpus2train(\"text8\")\n\nprint (\"number of examples in training set:\", len(trainset))\nprint(\"number of words:\", len(wordcount))"]},{"cell_type":"markdown","id":"682b0d21-23ec-4b5e-9de7-f322ed5a115e","metadata":{},"source":["## Dataset\n\n"]},{"cell_type":"markdown","id":"d311a127-fe2c-4c62-a532-79293547c4ff","metadata":{},"source":["A dataset is an object of a subclass of `torch.utils.data.Dataset`, which implements two methods: `__len__` which returns the number of examples in the dataset, and `__getitem__` which, given an integer *i*, returns the i<sup>th</sup> item of the dataset.\nWe can store additional information in the object. This is what we do here.\n\n"]},{"cell_type":"code","execution_count":1,"id":"17277f24-e049-4bbb-ade7-91c5fd7686c5","metadata":{},"outputs":[],"source":["#USE PYTORCH API for DATASETS\nimport torch\n\nclass W2VDataset(torch.utils.data.Dataset):\n  def __init__(self, file, window_size=16,voc_max_size=16384):\n      self.data, self.word2idx, self.idx2word, self.wordcount = corpus2train(file,voc_max_size)\n      self.window_size = window_size\n\n  def __len__(self):\n    return len(self.data)\n\n  def __getitem__(self, idx):\n    return self.data[idx]\n\ndataset= W2VDataset(\"text8\", window_size=4, voc_max_size=30000)"]},{"cell_type":"code","execution_count":1,"id":"d62f6284-a746-41f4-ad6a-14dcc8376131","metadata":{},"outputs":[],"source":["print (len(dataset.data))\nprint(len(dataset.wordcount))\n\nprint(dataset.wordcount[word2idx[\"man\"]])\nvisualizeWords = [\n    \"great\", \"cool\", \"brilliant\", \"wonderful\", \"well\", \"amazing\",\n    \"worth\", \"sweet\",\n    #\"enjoyable\",\n    \"boring\", \"bad\", \"dumb\",\n    #\"annoying\",\n    \"female\", \"male\", \"queen\", \"king\", \"man\", \"woman\", \"rain\", \"snow\",\n    \"hail\", \"coffee\", \"tea\"]\nfor w in visualizeWords:\n  print(w, dataset.wordcount[word2idx[w]])"]},{"cell_type":"markdown","id":"961267ae-fe79-4533-a817-5a085660f99c","metadata":{},"source":["## Evaluation\n\n"]},{"cell_type":"markdown","id":"430173da-43df-49d0-84de-2da516090b28","metadata":{},"source":["We will informally evaluate the learned model by displaying the vectors learned by the models for a few selected words.\n\nThe evaluation projects the vectors onto two dimensions (trying to minimize the distance between the original matrix and the projected \"deprojected\" matrix, cf. singular value decomposition algorithm), and creates the graph in the notebook.\n\nNote that if \\`evaluate\\` is called multiple times in the same cell, then the graphs overlap.\n\nNote, \\`evaluate\\` assumes that the implementation to be evaluated has two torch.nn.Embedding type variables, one named V which contains the center word vectors, and the other named U which contains the context word vectors.\n\nDon't forget to implement the end of the function!\n\n"]},{"cell_type":"code","execution_count":1,"id":"f686b5df-d355-4648-a48b-80073ebcaef9","metadata":{},"outputs":[],"source":["import numpy as np\nimport matplotlib\nmatplotlib.use('agg')\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom sklearn.manifold import TSNE\nimport pandas as pd\n\n\ndef evaluate(module, word2idx, idx2word, vistype=None):\n    #finalisation des vecteurs de mots\n    word_vectors = module.V.weight\n    #print(word_vectors.size())\n    word_vectors = word_vectors.detach().cpu()\n    wv = word_vectors.numpy()\n    wv = wv / np.linalg.norm(wv, axis=1, keepdims=True)\n\n    # we test on a subset of words (synonyms/antonyms...)\n    visualizeWords = [\n    \"great\", \"cool\", \"brilliant\", \"wonderful\", \"well\", \"amazing\",\n    \"worth\", \"sweet\", \"enjoyable\", \"boring\", \"bad\", \"dumb\",\n    \"annoying\", \"female\", \"male\", \"queen\", \"king\", \"man\", \"woman\", \"rain\", \"snow\",\n    \"hail\", \"coffee\", \"tea\"]\n    visualizeWords = [w for w in visualizeWords if w in word2idx]\n\n    visualizeIdx = [word2idx[word] for word in visualizeWords]\n    #print(visualizeIdx)\n    visualizeVecs = wv[visualizeIdx, :]\n\n    if vistype == 'tSNE':\n      tsne = TSNE(n_components=2)\n      X_tsne = tsne.fit_transform(wv)[visualizeIdx,:]\n      df = pd.DataFrame(X_tsne, index=visualizeWords, columns=['x', 'y'])\n\n      fig = plt.figure()\n      ax = fig.add_subplot(1, 1, 1)\n\n      ax.scatter(df['x'], df['y'])\n\n      for word, pos in df.iterrows():\n        ax.annotate(word, pos)\n\n      plt.show()\n\n    else:\n      # dimension reduction of vectors (on 2D for visualisation)\n      temp = (visualizeVecs - np.mean(visualizeVecs, axis=0))\n      covariance = 1.0 / len(visualizeIdx) * temp.T.dot(temp)\n      U,S,V = np.linalg.svd(covariance)\n      #print(U.shape, S.shape, V.shape)\n      coord = temp.dot(U[:,0:2])\n\n      for i in range(len(visualizeWords)):\n        plt.text(coord[i,0], coord[i,1], visualizeWords[i],\n            bbox=dict(facecolor='green', alpha=0.1))\n\n      plt.xlim((np.min(coord[:,0]), np.max(coord[:,0])))\n      plt.ylim((np.min(coord[:,1]), np.max(coord[:,1])))\n\n    #plt.savefig('word_vectors.png')\n\n\n    #TODO:\n    #Display the ten words most similar to \"man\"\n    # using \"cosine similarity\"\n    #https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html\n\n    # your code here"]},{"cell_type":"markdown","id":"186ffcfa-75c6-4a04-866a-c9750b48a27f","metadata":{},"source":["## The training loop\n\n"]},{"cell_type":"code","execution_count":1,"id":"48c23378-12f8-400c-879e-32ce0e3862ed","metadata":{},"outputs":[],"source":["#module : your w2v implementation\n#opt: the gradient descent optimizer\n#dataset: array containing all data\n#nb_epoch, the number of passes over the train_set\n\ndef train(module, opt, sched, loss_module, dataset, nb_epoch, batch_size):\n\n  dataloader = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True)\n  ignore_index = -100\n  max_index = len(dataset.wordcount)\n\n  for epoch in range(nb_epoch):\n    epoch_loss = 0\n\n    for batch_center in dataloader:\n      # your code here\n      \n\n    print(f\"epoch: {epoch}\\tloss: {epoch_loss/len(dataloader)}\")\n    sched.step(epoch_loss)\n    print(sched.get_last_lr())\n\n  evaluate(module, dataset.word2idx, dataset.idx2word)"]},{"cell_type":"code","execution_count":1,"id":"4391d82e-addb-4d9f-9712-ef89b82c044b","metadata":{},"outputs":[],"source":["class Word2Vec(torch.nn.Module):\n  def __init__(self, lexicon_size, vec_size, device):\n    super().__init__()\n    self.V = torch.nn.Embedding(lexicon_size, vec_size)\n    self.vec_size =vec_size\n    self.device=device\n    self.init_emb()\n    self = self.to(device)\n\n  # initialisation des embeddings comme Mikolov et al.\n  def init_emb(self):\n    self.V.weight.data.normal_(std= 0.5 / math.sqrt(self.vec_size))\n\n  # calcule les scores <w_o, w_c> pour tout w_o\n  def forward(self,idx_c, weight=None):\n    # your code here\n    \n\n    return torch.einsum('bi,li->bl',v,u)"]},{"cell_type":"code","execution_count":1,"id":"6a4afe28-4a1e-4dde-b0a6-14dcbb5c8b79","metadata":{},"outputs":[],"source":["#device =torch.device(\"mps\")\ndevice =torch.device(\"cuda:0\")\n#device =torch.device(\"cpu\")\n\nVECTOR_SIZE=100\n\nw2v = Word2Vec(len(dataset.idx2word), VECTOR_SIZE, device)\n\n\n# display vectors after random initialization\nevaluate(w2v, dataset.word2idx, dataset.idx2word)"]},{"cell_type":"code","execution_count":1,"id":"5deb5204-944e-447b-ba8f-9a598a96ff57","metadata":{},"outputs":[],"source":["class MLELoss(torch.nn.Module):\n  def __init__(self):\n    super().__init__()\n\n    # Loss function\n    self.ls = torch.nn.CrossEntropyLoss()\n\n  # computes scores\n  def forward(self, net, input, target, weight):\n    return self.ls(net(input, weight=weight), target)"]},{"cell_type":"markdown","id":"702d8a6e-bcd1-4810-8f3b-e71b67b31716","metadata":{},"source":["## Creation and training\n\n"]},{"cell_type":"code","execution_count":1,"id":"ac60b3d4-ee4a-41d2-9a85-002ba1652112","metadata":{},"outputs":[],"source":["BATCH_SIZE = 20000\n\noptimizer = torch.optim.RAdam(w2v.parameters(), lr=1e-2,decoupled_weight_decay=True, weight_decay=1e-5)\n# optimizer = torch.optim.SGD(w2v.parameters(), lr=1e2, weight_decay=1e-5)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=20)\n\nloss_module = MLELoss()\ntrain(w2v, optimizer, scheduler, loss_module, dataset, 1, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"2c7f698c-ad6d-433f-92a9-78f29774a0dc","metadata":{},"outputs":[],"source":["# measure time\n%timeit train(w2v, optimizer, scheduler, loss_module, dataset, 1, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"4f220016-c06f-4a63-abf7-cf3250ed671c","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 1, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"e12d7504-b350-496d-95e0-2302c7ece3f1","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 10, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"4d253156-fcb4-4a88-9f66-538d022ac71c","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 20, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"4acdfb9e-b3fa-448a-895c-9fddb2bb2e09","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 50, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"10470181-055a-42e4-82f4-c65ba91fee71","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 100, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"60a4da89-0e42-4465-9741-26995f1a1bb2","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 500, BATCH_SIZE)"]},{"cell_type":"code","execution_count":1,"id":"b94fbefa-74cd-41cf-9aaa-36fb93feb793","metadata":{},"outputs":[],"source":["train(w2v, optimizer, scheduler, loss_module, dataset, 1000, BATCH_SIZE)"]}],"metadata":{"org":{"AUTHOR":"Joseph Le Roux","DATE":"2025-11-19","DESCRIPTION":"Implementation of the skip-gram model (MLE)"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":5}