{"cells":[{"cell_type":"markdown","id":"6f965ef5-bf7c-4f42-817d-d16aa8665499","metadata":{},"source":"Convolutional Networks for Classification\n=========================================\n\n**Author:** Joseph Le Roux\n\n**Date:** 2025-11-19\n\n"},{"cell_type":"markdown","id":"45a94016-aacf-4e5e-942e-b42cbe2de364","metadata":{},"source":["- AUTHOR: Joseph Le Roux\n","- DATE: 2025-11-19\n","- DESCRIPTION: Convolutional Networks for Classification\n"]},{"cell_type":"markdown","id":"eb5d2160-77e1-46a3-8a62-736f50384018","metadata":{},"source":"## Table of Contents\n\n- [Data Retrieval](#data-retrieval)\n    - [Preprocessing](#Preprocessing)\n    - [Creation of Conversion Tables](#creation-of-conversion-tables)\n    - [From Text to Indices](#from-text-to-indices)\n    - [Dataset](#Dataset)\n    - [Neural Networks](#neural-networks)\n    - [Training Loop](#training-loop)\n- [Let's go!](#lets-go)\n"},{"cell_type":"markdown","id":"b1818c64-dd61-410c-bce0-be7b01c41bb4","metadata":{},"source":["## Data Retrieval:PROPERTIES:\n\n"]},{"cell_type":"code","execution_count":1,"id":"4df145e6-6a4b-41f4-ba6f-71dbb0fd89b1","metadata":{},"outputs":[],"source":["!wget https://lipn.fr/~leroux/INFO3_DL4NLP/classif_cnn_train_instances.txt\n!wget https://lipn.fr/~leroux/INFO3_DL4NLP/classif_cnn_train_classes.txt"]},{"cell_type":"code","execution_count":1,"id":"a2eeb119-a6fe-4543-b4c8-59554b72ee6f","metadata":{},"outputs":[],"source":["!sort -R classif_cnn_train_instances.txt | head\n!sort -R classif_cnn_train_classes.txt | head\n# careful, don't spend too much time looking at the tweets, it's crazy..."]},{"cell_type":"markdown","id":"cea711c2-144e-4b18-b9e7-da74a46cf0a3","metadata":{},"source":["### Preprocessing\n\n"]},{"cell_type":"markdown","id":"3dd6313d-7bda-49d2-a75d-5213390ddc16","metadata":{},"source":["We need to combine the two files\n\n-   the one containing the tweets and their identifiers\n-   the one associating identifiers and classes\n\nto get the association between class and tweet.\n\nWe take this opportunity to apply the following preprocessing, which you must implement (this is also NLP, and it changes tensors…):\n\n-   if a word consists only of digits, replace it with:\n\n    __NUM__\n\n-   if a word is an HTTP(S) URL, replace it with:\n\n    __URL__\n\n"]},{"cell_type":"code","execution_count":1,"id":"0f0bf47e-7693-4701-b01b-6b2ec3c01ed2","metadata":{},"outputs":[],"source":["def preprocess(file_name_instances, file_name_classes, replace_num=True, replace_url=True):\n\n  fi = open(file_name_instances)\n  # cut at tabs\n  instances = [ line.split(\"\\t\") for line in fi]\n  fi.close()\n  # cuts \" and \\n\n  instances = [ (id[1:-1], tweet[1:-2]) for (id,tweet) in instances]\n\n\n  if replace_num:\n    \n\n\n  if replace_url:\n    \n\n  fc = open(file_name_classes)\n  classes = [ line.split(\"|\") for line in fc]\n  # cut at \\n\n  classes = [ (id,cls[:-1]) for (id,cls) in classes]\n  # transform to dict\n  classes = dict(classes)\n\n  # now, link tweets to categories, forget IDs\n  instances = [ (tweet, classes[id]) for (id, tweet) in instances if id in classes]\n\n  return instances"]},{"cell_type":"code","execution_count":1,"id":"e2fc61da-2fb0-47e9-b499-35a6dc5dc3c7","metadata":{},"outputs":[],"source":["instances = preprocess(\"classif_cnn_train_instances.txt\", \"classif_cnn_train_classes.txt\")\nprint(instances[:10])"]},{"cell_type":"markdown","id":"b20481d1-13ae-4322-aafd-8e662a324c0b","metadata":{},"source":["### Creation of Conversion Tables:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"f83aeb73-fa45-418c-8b52-9c5f9900431e","metadata":{},"source":["From the preprocessed data, we create the tables that associate:\n\n-   each word to a unique integer (its index in the lexicon)\n-   each class to a unique integer\n-   each (index of) word to its number of occurrences in the text\n\nWe add a special word \"UNK\" which represents all words that might appear in test data but not in the training set.\n\n"]},{"cell_type":"code","execution_count":1,"id":"44337ccc-9219-40ff-a6cc-64c371e57778","metadata":{},"outputs":[],"source":["def create_dicts(text_instances, unk = \"__UNK__\"):\n  cls2id = {}\n  id2cls = []\n\n  word2id = {}\n  id2word = []\n\n  wordidfreq = []\n\n  for (tweet, cls) in text_instances:\n\n    if cls not in cls2id:\n      cls2id[cls] = len(id2cls)\n      id2cls.append(cls)\n\n    tokens = tweet.split(\" \")\n    for token in tokens:\n      if token in word2id:\n        wordidfreq[word2id[token]] +=1\n      else:\n        word2id[token] = len(id2word)\n        id2word.append(token)\n        wordidfreq.append(1)\n\n\n  #add an extra token  for out-of-vocabulary words\n  word2id[unk] = len(id2word)\n  id2word.append(unk)\n\n  dicts = { \"cls2id\": cls2id,\n          \"id2cls\": id2cls,\n          \"word2id\": word2id,\n          \"id2word\": id2word,\n           \"wordidfreq\": wordidfreq}\n\n  return dicts"]},{"cell_type":"code","execution_count":1,"id":"32139368-061c-4091-859e-8ba41defc9a1","metadata":{},"outputs":[],"source":["dicts = create_dicts(instances)"]},{"cell_type":"code","execution_count":1,"id":"004cccfa-cbe9-422c-9c31-d61dc2bcb11f","metadata":{},"outputs":[],"source":["#dicts['cls2id']\n#dicts['id2cls']\n#dicts['word2id']"]},{"cell_type":"markdown","id":"a20e9df0-0f7c-48cb-9323-c89649b47d14","metadata":{},"source":["### From Text to Indices:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"47b7dbdd-be65-499b-bb79-29ec61b5f043","metadata":{},"source":["we replace all strings with the integers that represent them\n\nOur training (and development) data will be represented by a list of tensor pairs, one for the class, one for the tweet\n\n"]},{"cell_type":"code","execution_count":1,"id":"87ffa9ce-6995-45ae-9048-72573949be19","metadata":{},"outputs":[],"source":["import torch\n# create a list (category tensor, word indexes tensor)\n#(be careful: integer tensors)\ndef instances_to_pytorch(instances, dicts,device):"]},{"cell_type":"code","execution_count":1,"id":"b574e050-f25e-4e93-b270-82da791fd4f2","metadata":{},"outputs":[],"source":["tinstances = instances_to_pytorch(instances, dicts, device=\"cuda:0\")\nprint(tinstances[:10])"]},{"cell_type":"markdown","id":"dde47de4-54df-419e-829c-5092efdbf110","metadata":{},"source":["### Dataset\n\n"]},{"cell_type":"markdown","id":"da92f5bb-45a1-4290-93cd-8a2bbb268ce2","metadata":{},"source":["Implement methods for the subclass of  `Dataset`  used to go trough the data.\n\nIn the constructor,  argument `threshold` indicates the minimal nu,ber of occurrences for a word in the corpus to be part of the dataset. If it appears less than `threshold` times, it must be removed and replaced by `__UNK__`.\n\n"]},{"cell_type":"code","execution_count":1,"id":"497c1e7e-d7bb-4e02-bdc8-21ef2aacc44a","metadata":{},"outputs":[],"source":["class TweetDataset(torch.utils.data.Dataset):\n  def __init__(self, instances, dicts, device, threshold=0) -> None:\n    super().__init__()\n\n    self.tinstances = instances_to_pytorch(instances, dicts, device)\n    if threshold > 0:\n      # replace rare words by __UNK__\n                  \n\n  def __len__(self):\n    \n\n  def __getitem__(self, idx):\n    \n\n# create instances\ninstances = preprocess(\"classif_cnn_train_instances.txt\", \"classif_cnn_train_classes.txt\")\nimport random\nrandom.shuffle(instances)\n\n# 90% train / 10% eval\ntrain_instances = instances[:int(0.9 * len(instances))]\ndev_instances = instances[int(0.9 * len(instances)):]\ndicts = create_dicts(train_instances)\n\n# follow pytorch API\ndevice = torch.device(\"cuda:0\")\ntrain_dataset = TweetDataset(train_instances, dicts, device, threshold=2)\ndev_dataset = TweetDataset(dev_instances, dicts, device)"]},{"cell_type":"markdown","id":"16984cfc-a6dc-4c06-a79f-b743ba98ed8a","metadata":{},"source":["### Neural Networks:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"6ae059da-c62a-4998-a5ce-95b9f4198553","metadata":{},"source":["You will implement:\n\n1.  An MLP consisting of two linear transformations separated by an activation, with Dropout during training.\n2.  A network that calculates convolutions of sizes 2, 3, 4, 5, 6, 7 with max pooling. The resulting vectors for each size are summed and the result is returned (to *bootstrap* your model you can start with a single convolution of size 3)\n3.  A network that implements the classifier. It contains:\n\n4.  a word vector table\n5.  a convolutional module to transform inputs into a fixed-size vector\n6.  a dropout applied to the output of the previous layer\n7.  an MLP module for classification.\n\nFor the forward pass which takes a list *l* of (class, tweet) pairs, the difficulty for this module is constructing the batches as a tensor before passing them to the sub-networks.\n\n"]},{"cell_type":"markdown","id":"23c52d19-7237-4da6-83ea-6bdffd6c992b","metadata":{},"source":["#### If we do not use padding:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"4473281c-f26f-45a2-82e5-7d79a2931c8a","metadata":{},"source":["If *l* is of length *t* and the longest tweet has length *m*, we must construct a tensor of size *(l,m,word vector size)*. Empty positions are initialized with zero vectors.\n\n"]},{"cell_type":"markdown","id":"6d611398-1070-4996-a436-7f468716cac2","metadata":{},"source":["#### with padding of size *k*:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"395eea5b-48b7-4ccb-be44-b84a042b2417","metadata":{},"source":["We will construct a tensor of size *(l,m+2k,vector size)*. Pay attention to the placement of word vectors in this tensor.\n\n"]},{"cell_type":"markdown","id":"38945d1c-d8dd-4d1d-b846-9df3106080a3","metadata":{},"source":["#### What to do?:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"84ffbcb0-7855-41f6-9206-f3b1d87f2a34","metadata":{},"source":["Of course, you are strongly encouraged to implement padding!\n\n"]},{"cell_type":"code","execution_count":1,"id":"5d05c5f4-9255-4f1e-994c-9380ced83ffb","metadata":{},"outputs":[],"source":["class MLP(torch.nn.Module):\n  def __init__(self, in_dim, hidden_dim, out_dim, activation, dropout=0.5):\n    super(MLP,self).__init__()\n\n    \n\n\n  def forward(self, input):\n    \n\nclass FeatureConv(torch.nn.Module):\n  def __init__(self, word_emb_dim, conv_dim):\n    super(FeatureConv,self).__init__()\n\n\n    \n\n\n\n    # not practical, use ModuleList \n    #self.conv2 = ...\n    #self.conv3 = ...\n    #self.conv4 = ...\n    #self.conv5 = ...     your code\n    #self.conv6 = ...\n    #self.conv7 = ...\n    #self.pool = ...\n\n    \n\n\n  def forward(self, x):\n    # pool the sum of convolutions\n    \n\nclass Classifier(torch.nn.Module):\n  def __init__(self, lexicon_size, word_emb_dim, conv_dim, hidden_dim, nb_classes, dropout=0.5):\n    super(Classifier,self).__init__()  \n\n    \n\n  def forward(self, x):\n    \n    return x"]},{"cell_type":"markdown","id":"c6f942e3-a2d2-4db0-aa18-70d901dedd3f","metadata":{},"source":["### Training Loop:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"1a54e597-5e6c-464a-8a81-bd9d9c18be40","metadata":{},"source":["You must implement:\n\n1.  the evaluate function which makes a prediction for each example in the passed instances list, and returns the ratio of correct answers.\n2.  the train function which implements stochastic gradient descent learning in batches (with batch). Note that, to improve system robustness, during training we randomly replace each occurrence of a word *w* with the \"UNK\" token defined above with a probability of $\\frac{\\alpha}{\\alphac+ \\#w}$ where $\\#w$) is the number of occurrences of $w$) in the corpus.\n\n"]},{"cell_type":"code","execution_count":1,"id":"51f2abbb-20a3-4d78-8b89-8788d8ba9b13","metadata":{},"outputs":[],"source":["print(dicts['wordidfreq'][3])"]},{"cell_type":"code","execution_count":1,"id":"28bd676d-33db-46b6-8a5f-46f21cc03924","metadata":{},"outputs":[],"source":["import random\nimport numpy as np\n\n# a new index for pads\npad_index = dicts['word2id']['__UNK__'] + 1\n\n\ndef evaluate(data_loader, net):\n  net.eval()\n  correct = 0\n  all = 0\n  for batch_input, batch_output in data_loader:\n\n    sols = torch.max(net(batch_input), dim=1)[1]\n    #print(sols.size())\n    correct += torch.eq(sols, batch_output).sum()\n    all += batch_output.size(0)\n\n  acc = correct/all\n  print(f\"Dev Accuracy: {acc}\")\n  return acc\n\n\n# takes a list of pairs (category tensor, word indexes tensor) for each tweet\n# transform it to a pair (categories tensor, word indexes tensor) for the whole batch\ndef collate_to_same_length(batch):\n  list_input = []\n  list_output = []\n  max_len = 0\n  for (input,output) in batch:\n    \n\n  return (input, output)\n\n\n\n\n\n\ndef train(train_dataset, dev_dataset, dicts, nb_epochs, batch_size, net, opt, sched=None, alpha=1.0):\n\n  unk_index = len(dicts[\"id2word\"]) - 1\n  loss_func = torch.nn.CrossEntropyLoss()\n\n\n\n  train_dl = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, \n                                         shuffle=True,\n                                         collate_fn=collate_to_same_length)\n\n  dev_dl = torch.utils.data.DataLoader(dataset=dev_dataset, batch_size=batch_size*2, \n                                       shuffle=False,\n                                       collate_fn=collate_to_same_length)\n\n  for e in range(nb_epochs):\n    net.train()\n    epoch_loss = 0.0\n\n    for batch_input, batch_output in train_dl:\n\n\n\n      pred = net(batch_input)\n      #print(pred.size(), batch_output.size())\n      loss = loss_func(pred, batch_output)\n      epoch_loss += loss.item()\n\n      loss.backward()\n      opt.step()\n      opt.zero_grad()\n\n    print(f\"epoch: {e}\\tloss: {32 * epoch_loss/len(train_dataset)}\")\n    evaluate(dev_dl, net)"]},{"cell_type":"markdown","id":"38b3c5a8-db0f-420c-8b65-e8e19c734ca4","metadata":{},"source":["## Let's go!:PROPERTIES:\n\n"]},{"cell_type":"markdown","id":"8c2e0ca5-7639-4b65-9b41-b6d8583a8177","metadata":{},"source":["We put everything together in the following script: you should achieve ~63% correct answers on the dev set\n\n"]},{"cell_type":"code","execution_count":1,"id":"5ee68abc-ccd6-4091-bff1-a4da521700a5","metadata":{},"outputs":[],"source":["classifier = Classifier(len(dicts[\"id2word\"]), 100, 400, 100, 4).to(device)\noptimizer = torch.optim.Adam(classifier.parameters(), lr=0.001)\n#optimizer = torch.optim.SGD(classifier.parameters(), lr=0.001, momentum=0.9, nesterov=True)\n\n# should reach 63% on dev\nscheduler = None # use a  scheduler to find better step\ntrain(train_dataset, dev_dataset, dicts, 40, 16, classifier, optimizer, scheduler)"]}],"metadata":{"org":{"AUTHOR":"Joseph Le Roux","DATE":"2025-11-19","DESCRIPTION":"Convolutional Networks for Classification"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":5}