{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Exercise Set 1\n", "

IN4080 - 2025

\n", "\n", "*You will probably not manage to work through all of this exercise set during the group session. Continue\n", "to work on it by yourself after the group session and return to the teacher in later group sessions if you\n", "have any questions.*\n", "\n", "\n", "This exercises requires the following packages:\n", "* Spacy \n", "* Numpy \n", "* Pandas\n", "* Matplotlib" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 0: Set up a working environment on your PC\n", "\n", "Follow the installation instructions on the course web page to set up a working environment on your\n", "own computer.\n", "The following exercises should be solved interactively in Python. We recommend using a Jupyter\n", "Notebook, but you can also work in a standard interactive Python prompt." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Jupyter notebooks? \n", " The Jupyter Notebook is an interactive computing environment. \n", "\n", " Read about: \n", " - [Notebook Basics](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Notebook%20Basics.html) \n", " - [Running Code](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Running%20Code.html) \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Precpocessing Data with SpaCy\n", "\n", "We will work with the book *Peter Pan*, which is available as plain text on Project Gutenberg. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "a) Download the Plain Text UTF-8 file from https://www.gutenberg.org/ebooks/16 and place it in your working\n", "directory. Rename the file if necessary. Open the file in a text editor. Are there any particularities in the document \n", "that you’ll need to watch out when processing?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer with text here:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "b) This is a Jupyter notebook which is organized into cells. There are essentially two types of cells: text (Markdown)\n", "cells and code cells. You can execute a code cell by clicking on the Play button or by pressing Ctrl+Enter.\n", "Create a code cell below, load in the Peter Pan text, and print the length of the text. What does this number tell you? \n", "\n", "\n", "
\n", " Code Solution\n", "\n", "```python\n", "with open(‘peterpan.txt’, ‘r’, encoding=’utf-8-sig’) as f:\n", " text = f.read()\n", " print(len(text))\n", "```\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "c) We would now like preprocess the raw text. The NLP pipeline from `SpaCy` does a lot of the heavy lifting for us.\n", "Run the text through the SpaCy pipeline and print out the number of sentences in the resulting `Doc` object.\n", "Does the number of sentences correspond to your expectations? Inspect the data.\n", "\n", "[SpaCy Usage Documentation](https://spacy.io/usage)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "d) It turns out that the initial text contains a lot of line breaks that make the processing harder than\n", "necessary. Replace all line breaks in the raw text with a space. \n", "\n", "*Hint* use regular expressions and the `re` library.\n", "\n", "If you are not familiar with regular expressions, let your teachers know. We won’t need them now, but they \n", "can come quite handy for a lot of text processing tasks, so it’s worth investing some time in learning the basics…" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "e) Run the new text through the sentence tokenizer again. How do you judge the result? Have new errors been introduced?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Frequency distributions and Pandas\n", "\n", "In this exercise set, we want to study the frequency distributions of words in the text.\n", "Python provides a Counter class that counts repeated elements in a list and stores the counts as a\n", "dictionary. We can set up the counter, get the most common elements, and the count for a specific element as follows:\n", "\n", "```python\n", "import collections\n", "\n", "c = collections.Counter(tokens)\n", "print(c.most_common(10))\n", "print(c['with'])\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "a) Print out the 10 most frequent tokens in the text.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "b) You will see that punctuation marks are among the most frequent items in the result. Remove them from the counter.\n", "\n", "
\n", " Hints \n", "\t\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "c) It would be nice if we could display the frequency distribution as a nicely formatted table. Let us use a\n", "DataFrame from the Pandas package for this. In general, it is easy to populate a DataFrame with the\n", "contents of a Python dictionary, as in the following code snippet:\n", "\n", "```python\n", "import pandas as pd\n", "\n", "d = {‘apple’: 5, ‘orange’: 8, ‘banana’: 51, ‘strawberry’: 20}\n", "df = pd.DataFrame(d.items(), columns=['fruit', 'number'])\n", "\n", "df.head() # Display the first rows\n", "```\n", "\n", "You can do the same with your word counter. \n", "\n", "How many types and how many tokens does the text contain? What is its type-token-ratio?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "d) With Pandas, you can easily select rows of a dataframe according to a particular criterion. For example,\n", "this command displays all words that occur ten times or more:\n", "\n", "```python\n", "df[df[‘count’] >= 10]\n", "```\n", "\n", "How many hapaxes are in the dataset? What percentage of all word types are hapaxes?\n", "How many word types start with upper case A?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: Plotting with Matplotlib\n", "\n", "Matplotlib is a package for making plots and figures in Python. When using Jupyter notebooks, the plots\n", "are directly displayed in the notebook. This code snippet generates a simple plot:\n", "\n", "```python \n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "numbers = np.arange(10)\n", "print(numbers)\n", "plt.plot(numbers)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "a) Let us create a plot from our frequency distribution. You can directly use the `plot()` method of the\n", "dataframe as follows:\n", "\n", "```python\n", "df = df.sort_values('count', ascending=False)\n", "df.plot(x='word', y='count')\n", "```\n", "\n", "Does the result of this plot correspond to your expectations? " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "b) Let's make some more visualizations. \n", "\n", "* Modify the command to only display the 20 most frequent words. \n", "* Try to display all words on the x-axis.\n", "* For frequency plots, it is more natural to use bar charts. Switch the type of the plot with the `kind='bar'` parameter." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "c) Zipf’s law states that the product of the frequency of a word and of its rank is approximately constant.\n", "Let us verify this law on a subset of our frequency distribution. Select the 2000 first words of the\n", "sorted frequency table. Reset the indices in the resulting dataframe so that we can use each index for the rank. Now, add a new column with the product of rank and frequency.\n", "\n", "
\n", "Hints\n", "\n", "```python\n", "\tdf_zipf = df[:2000].reset_index(drop=True) \n", "\tdf_zipf['z'] = df_zipf.index.values * df_zipf['Count']\n", "```\n", "
\n", "\n", "What are the highest, lowest and average values of z that you observe? Plot the z values as a line chart." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 4 }