# Load packages
import requests
import pymupdf
# Download the data
url = "https://funnyengwish.wordpress.com/wp-content/uploads/2017/05/pratchett_terry_wyrd_sisters_-_royallib_ru.pdf"
response = requests.get(url)
# Extract data from pdf
data = response.content
doc = pymupdf.Document(stream=data)
# Create text from first pdf page
page1 = doc[0].get_text()Text processing
In this section, we will use the TextBlob package for part of speech tagging and basic tokenization.
TextBlob
TextBlob is the NLP package that we will use in this course for tagging, tokenization, normalization, and sentiment analysis.
We first need to load it in our session:
from textblob import TextBlobBefore we can use TextBlob on our text, we need to convert the page1 string into a TextBlob object:
text = TextBlob(page1)
type(text)textblob.blob.TextBlobPart of speech tagging
Part of speech tagging attributes parts of speech (POS) tags to each word of a text.
You can do this simply by using the tags property on a TextBlob object: text.tags. Because there are a lot of words in the first pdf page, this would create a very long output.
The result is a list:
type(text.tags)--------------------------------------------------------------------------- LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:46, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 45 try: ---> 46 return func(*args, **kwargs) 47 except LookupError as error: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/taggers.py:35, in NLTKTagger.tag(self, text) 33 text = tb.TextBlob(text) ---> 35 return nltk.tag.pos_tag(text.tokens) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/__init__.py:168, in pos_tag(tokens, tagset, lang) 144 """ 145 Use NLTK's currently recommended part of speech tagger to 146 tag the given list of tokens. (...) 166 :rtype: list(tuple(str, str)) 167 """ --> 168 tagger = _get_tagger(lang) 169 return _pos_tag(tokens, tagset, tagger, lang) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/__init__.py:110, in _get_tagger(lang) 109 else: --> 110 tagger = PerceptronTagger() 111 return tagger File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/perceptron.py:183, in PerceptronTagger.__init__(self, load, lang, loc) 182 if load: --> 183 self.load_from_json(lang, loc) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/perceptron.py:279, in PerceptronTagger.load_from_json(self, lang, loc) 278 if loc is None: --> 279 loc = find(f"taggers/averaged_perceptron_tagger_{lang}/") 280 elif isinstance(loc, str): 281 # Backward compatible: 282 # - absolute paths are explicit filesystem locations 283 # - relative strings are treated as NLTK resource names and resolved via find() File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'averaged_perceptron_tagger_eng' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('averaged_perceptron_tagger_eng') For more information see: https://www.nltk.org/data.html Attempted to load 'taggers/averaged_perceptron_tagger_eng/' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** The above exception was the direct cause of the following exception: MissingCorpusError Traceback (most recent call last) Cell In[4], line 1 ----> 1 type(text.tags) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:517, in BaseBlob.pos_tags(self) 498 """Returns an list of tuples of the form (word, POS tag). 499 500 Example: (...) 512 :rtype: list of tuples 513 """ 514 if isinstance(self, TextBlob): 515 return [ 516 val --> 517 for sublist in [s.pos_tags for s in self.sentences] 518 for val in sublist 519 ] 520 else: 521 return [ 522 (Word(str(word), pos_tag=t), str(t)) 523 for word, t in self.pos_tagger.tag(self) 524 if not PUNCTUATION_REGEX.match(str(t)) 525 ] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:523, in BaseBlob.pos_tags(self) 515 return [ 516 val 517 for sublist in [s.pos_tags for s in self.sentences] 518 for val in sublist 519 ] 520 else: 521 return [ 522 (Word(str(word), pos_tag=t), str(t)) --> 523 for word, t in self.pos_tagger.tag(self) 524 if not PUNCTUATION_REGEX.match(str(t)) 525 ] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:48, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 46 return func(*args, **kwargs) 47 except LookupError as error: ---> 48 raise MissingCorpusError() from error MissingCorpusError: Looks like you are missing some required data for this feature. To download the necessary data, simply run python -m textblob.download_corpora or use the NLTK downloader to download the missing data: http://nltk.org/data.html If this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.
And each element of the list is a tuple:
type(text.tags[0])--------------------------------------------------------------------------- LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:46, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 45 try: ---> 46 return func(*args, **kwargs) 47 except LookupError as error: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/taggers.py:35, in NLTKTagger.tag(self, text) 33 text = tb.TextBlob(text) ---> 35 return nltk.tag.pos_tag(text.tokens) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/__init__.py:168, in pos_tag(tokens, tagset, lang) 144 """ 145 Use NLTK's currently recommended part of speech tagger to 146 tag the given list of tokens. (...) 166 :rtype: list(tuple(str, str)) 167 """ --> 168 tagger = _get_tagger(lang) 169 return _pos_tag(tokens, tagset, tagger, lang) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/__init__.py:110, in _get_tagger(lang) 109 else: --> 110 tagger = PerceptronTagger() 111 return tagger File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/perceptron.py:183, in PerceptronTagger.__init__(self, load, lang, loc) 182 if load: --> 183 self.load_from_json(lang, loc) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/perceptron.py:279, in PerceptronTagger.load_from_json(self, lang, loc) 278 if loc is None: --> 279 loc = find(f"taggers/averaged_perceptron_tagger_{lang}/") 280 elif isinstance(loc, str): 281 # Backward compatible: 282 # - absolute paths are explicit filesystem locations 283 # - relative strings are treated as NLTK resource names and resolved via find() File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'averaged_perceptron_tagger_eng' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('averaged_perceptron_tagger_eng') For more information see: https://www.nltk.org/data.html Attempted to load 'taggers/averaged_perceptron_tagger_eng/' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** The above exception was the direct cause of the following exception: MissingCorpusError Traceback (most recent call last) Cell In[5], line 1 ----> 1 type(text.tags[0]) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:517, in BaseBlob.pos_tags(self) 498 """Returns an list of tuples of the form (word, POS tag). 499 500 Example: (...) 512 :rtype: list of tuples 513 """ 514 if isinstance(self, TextBlob): 515 return [ 516 val --> 517 for sublist in [s.pos_tags for s in self.sentences] 518 for val in sublist 519 ] 520 else: 521 return [ 522 (Word(str(word), pos_tag=t), str(t)) 523 for word, t in self.pos_tagger.tag(self) 524 if not PUNCTUATION_REGEX.match(str(t)) 525 ] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:523, in BaseBlob.pos_tags(self) 515 return [ 516 val 517 for sublist in [s.pos_tags for s in self.sentences] 518 for val in sublist 519 ] 520 else: 521 return [ 522 (Word(str(word), pos_tag=t), str(t)) --> 523 for word, t in self.pos_tagger.tag(self) 524 if not PUNCTUATION_REGEX.match(str(t)) 525 ] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:48, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 46 return func(*args, **kwargs) 47 except LookupError as error: ---> 48 raise MissingCorpusError() from error MissingCorpusError: Looks like you are missing some required data for this feature. To download the necessary data, simply run python -m textblob.download_corpora or use the NLTK downloader to download the missing data: http://nltk.org/data.html If this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.
We don’t have to print the full list. Let’s only print the first 20 tuples:
text.tags[:20]--------------------------------------------------------------------------- LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:46, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 45 try: ---> 46 return func(*args, **kwargs) 47 except LookupError as error: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/taggers.py:35, in NLTKTagger.tag(self, text) 33 text = tb.TextBlob(text) ---> 35 return nltk.tag.pos_tag(text.tokens) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/__init__.py:168, in pos_tag(tokens, tagset, lang) 144 """ 145 Use NLTK's currently recommended part of speech tagger to 146 tag the given list of tokens. (...) 166 :rtype: list(tuple(str, str)) 167 """ --> 168 tagger = _get_tagger(lang) 169 return _pos_tag(tokens, tagset, tagger, lang) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/__init__.py:110, in _get_tagger(lang) 109 else: --> 110 tagger = PerceptronTagger() 111 return tagger File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/perceptron.py:183, in PerceptronTagger.__init__(self, load, lang, loc) 182 if load: --> 183 self.load_from_json(lang, loc) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/tag/perceptron.py:279, in PerceptronTagger.load_from_json(self, lang, loc) 278 if loc is None: --> 279 loc = find(f"taggers/averaged_perceptron_tagger_{lang}/") 280 elif isinstance(loc, str): 281 # Backward compatible: 282 # - absolute paths are explicit filesystem locations 283 # - relative strings are treated as NLTK resource names and resolved via find() File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'averaged_perceptron_tagger_eng' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('averaged_perceptron_tagger_eng') For more information see: https://www.nltk.org/data.html Attempted to load 'taggers/averaged_perceptron_tagger_eng/' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** The above exception was the direct cause of the following exception: MissingCorpusError Traceback (most recent call last) Cell In[6], line 1 ----> 1 text.tags[:20] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:517, in BaseBlob.pos_tags(self) 498 """Returns an list of tuples of the form (word, POS tag). 499 500 Example: (...) 512 :rtype: list of tuples 513 """ 514 if isinstance(self, TextBlob): 515 return [ 516 val --> 517 for sublist in [s.pos_tags for s in self.sentences] 518 for val in sublist 519 ] 520 else: 521 return [ 522 (Word(str(word), pos_tag=t), str(t)) 523 for word, t in self.pos_tagger.tag(self) 524 if not PUNCTUATION_REGEX.match(str(t)) 525 ] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:523, in BaseBlob.pos_tags(self) 515 return [ 516 val 517 for sublist in [s.pos_tags for s in self.sentences] 518 for val in sublist 519 ] 520 else: 521 return [ 522 (Word(str(word), pos_tag=t), str(t)) --> 523 for word, t in self.pos_tagger.tag(self) 524 if not PUNCTUATION_REGEX.match(str(t)) 525 ] File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:48, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 46 return func(*args, **kwargs) 47 except LookupError as error: ---> 48 raise MissingCorpusError() from error MissingCorpusError: Looks like you are missing some required data for this feature. To download the necessary data, simply run python -m textblob.download_corpora or use the NLTK downloader to download the missing data: http://nltk.org/data.html If this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.
Noun phrases extraction
Noun phrases can be extracted with the noun_phrases property:
print(text.noun_phrases)--------------------------------------------------------------------------- LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:85, in LazyCorpusLoader.__load(self) 84 try: ---> 85 root = nltk.data.find(f"{self.subdir}/{zip_name}") 86 except LookupError: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'brown' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('brown') For more information see: https://www.nltk.org/data.html Attempted to load 'corpora/brown.zip/brown/' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** During handling of the above exception, another exception occurred: LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:46, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 45 try: ---> 46 return func(*args, **kwargs) 47 except LookupError as error: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/np_extractors.py:115, in FastNPExtractor.train(self) 113 @requires_nltk_corpus 114 def train(self): --> 115 train_data = nltk.corpus.brown.tagged_sents(categories="news") 116 regexp_tagger = nltk.RegexpTagger( 117 [ 118 (r"^-?[0-9]+(.[0-9]+)?$", "CD"), (...) 130 ] 131 ) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:129, in LazyCorpusLoader.__getattr__(self, attr) 125 raise AttributeError( 126 f"{type(self).__name__} object has no attribute {attr!r}" 127 ) --> 129 self.__load() 130 # This looks circular, but its not, since __load() changes our 131 # __class__ to something new: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:87, in LazyCorpusLoader.__load(self) 86 except LookupError: ---> 87 raise e 89 # Load the corpus. File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:82, in LazyCorpusLoader.__load(self) 81 try: ---> 82 root = nltk.data.find(f"{self.subdir}/{self.__name}") 83 except LookupError as e: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'brown' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('brown') For more information see: https://www.nltk.org/data.html Attempted to load 'corpora/brown' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** The above exception was the direct cause of the following exception: MissingCorpusError Traceback (most recent call last) Cell In[7], line 1 ----> 1 print(text.noun_phrases) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:491, in BaseBlob.noun_phrases(self) 485 @cached_property 486 def noun_phrases(self): 487 """Returns a list of noun phrases for this blob.""" 488 return WordList( 489 [ 490 phrase.strip().lower() --> 491 for phrase in self.np_extractor.extract(self.raw) 492 if len(phrase) > 1 493 ] 494 ) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/np_extractors.py:145, in FastNPExtractor.extract(self, text) 143 """Return a list of noun phrases (strings) for body of text.""" 144 if not self._trained: --> 145 self.train() 146 tokens = self._tokenize_sentence(text) 147 tagged = self.tagger.tag(tokens) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:48, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 46 return func(*args, **kwargs) 47 except LookupError as error: ---> 48 raise MissingCorpusError() from error MissingCorpusError: Looks like you are missing some required data for this feature. To download the necessary data, simply run python -m textblob.download_corpora or use the NLTK downloader to download the missing data: http://nltk.org/data.html If this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.
The output is a WordList object:
type(text.noun_phrases)--------------------------------------------------------------------------- LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:85, in LazyCorpusLoader.__load(self) 84 try: ---> 85 root = nltk.data.find(f"{self.subdir}/{zip_name}") 86 except LookupError: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'brown' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('brown') For more information see: https://www.nltk.org/data.html Attempted to load 'corpora/brown.zip/brown/' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** During handling of the above exception, another exception occurred: LookupError Traceback (most recent call last) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:46, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 45 try: ---> 46 return func(*args, **kwargs) 47 except LookupError as error: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/np_extractors.py:115, in FastNPExtractor.train(self) 113 @requires_nltk_corpus 114 def train(self): --> 115 train_data = nltk.corpus.brown.tagged_sents(categories="news") 116 regexp_tagger = nltk.RegexpTagger( 117 [ 118 (r"^-?[0-9]+(.[0-9]+)?$", "CD"), (...) 130 ] 131 ) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:129, in LazyCorpusLoader.__getattr__(self, attr) 125 raise AttributeError( 126 f"{type(self).__name__} object has no attribute {attr!r}" 127 ) --> 129 self.__load() 130 # This looks circular, but its not, since __load() changes our 131 # __class__ to something new: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:87, in LazyCorpusLoader.__load(self) 86 except LookupError: ---> 87 raise e 89 # Load the corpus. File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/corpus/util.py:82, in LazyCorpusLoader.__load(self) 81 try: ---> 82 root = nltk.data.find(f"{self.subdir}/{self.__name}") 83 except LookupError as e: File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/nltk/data.py:738, in find(resource_name, paths) 737 resource_not_found = f"\n{sep}\n{msg}\n{sep}\n" --> 738 raise LookupError(resource_not_found) LookupError: ********************************************************************** Resource 'brown' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('brown') For more information see: https://www.nltk.org/data.html Attempted to load 'corpora/brown' Searched in: - '/home/marie/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/share/nltk_data' - '/home/marie/parvus/prog/mint/python/.venv/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' ********************************************************************** The above exception was the direct cause of the following exception: MissingCorpusError Traceback (most recent call last) Cell In[8], line 1 ----> 1 type(text.noun_phrases) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:32, in cached_property.__get__(self, obj, cls) 30 if obj is None: 31 return self ---> 32 value = obj.__dict__[self.func.__name__] = self.func(obj) 33 return value File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/blob.py:491, in BaseBlob.noun_phrases(self) 485 @cached_property 486 def noun_phrases(self): 487 """Returns a list of noun phrases for this blob.""" 488 return WordList( 489 [ 490 phrase.strip().lower() --> 491 for phrase in self.np_extractor.extract(self.raw) 492 if len(phrase) > 1 493 ] 494 ) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/en/np_extractors.py:145, in FastNPExtractor.extract(self, text) 143 """Return a list of noun phrases (strings) for body of text.""" 144 if not self._trained: --> 145 self.train() 146 tokens = self._tokenize_sentence(text) 147 tagged = self.tagger.tag(tokens) File ~/parvus/prog/mint/python/.venv/lib/python3.14/site-packages/textblob/decorators.py:48, in requires_nltk_corpus.<locals>.decorated(*args, **kwargs) 46 return func(*args, **kwargs) 47 except LookupError as error: ---> 48 raise MissingCorpusError() from error MissingCorpusError: Looks like you are missing some required data for this feature. To download the necessary data, simply run python -m textblob.download_corpora or use the NLTK downloader to download the missing data: http://nltk.org/data.html If this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.
Tokenization
Words
TextBlob allows you to extract words easily with the words attribute:
print(text.words)['Terry', 'Pratchett', 'Wyrd', 'Sisters', 'Starring', 'Three', 'Witches', 'also', 'kings', 'daggers', 'crowns', 'storms', 'dwarfs', 'cats', 'ghosts', 'spectres', 'apes', 'bandits', 'demons', 'forests', 'heirs', 'jesters', 'tortures', 'trolls', 'turntables', 'general', 'rejoicing', 'and', 'drivers', 'alarums', 'The', 'wind', 'howled', 'Lightning', 'stabbed', 'at', 'the', 'earth', 'erratically', 'like', 'an', 'inefficient', 'assassin', 'Thunder', 'rolled', 'back', 'and', 'forth', 'across', 'the', 'dark', 'rain-lashed', 'hills', 'The', 'night', 'was', 'as', 'black', 'as', 'the', 'inside', 'of', 'a', 'cat', 'It', 'was', 'the', 'kind', 'of', 'night', 'you', 'could', 'believe', 'on', 'which', 'gods', 'moved', 'men', 'as', 'though', 'they', 'were', 'pawns', 'on', 'the', 'chessboard', 'of', 'fate', 'In', 'the', 'middle', 'of', 'this', 'elemental', 'storm', 'a', 'fire', 'gleamed', 'among', 'the', 'dripping', 'furze', 'bushes', 'like', 'the', 'madness', 'in', 'a', 'weasel', "'s", 'eye', 'It', 'illuminated', 'three', 'hunched', 'figures', 'As', 'the', 'cauldron', 'bubbled', 'an', 'eldritch', 'voice', 'shrieked', "'When", 'shall', 'we', 'three', 'meet', 'again', 'There', 'was', 'a', 'pause', 'Finally', 'another', 'voice', 'said', 'in', 'far', 'more', 'ordinary', 'tones', "'Well", 'I', 'can', 'do', 'next', 'Tuesday', 'Through', 'the', 'fathomless', 'deeps', 'of', 'space', 'swims', 'the', 'star', 'turtle', 'Great', "A'Tuin", 'bearing', 'on', 'its', 'back', 'the', 'four', 'giant', 'elephants', 'who', 'carry', 'on', 'their', 'shoulders', 'the', 'mass', 'of', 'the', 'Discworld', 'A', 'tiny', 'sun', 'and', 'moon', 'spin', 'around', 'them', 'on', 'a', 'complicated', 'orbit', 'to', 'induce', 'seasons', 'so', 'probably', 'nowhere', 'else', 'in', 'the', 'multiverse', 'is', 'it', 'sometimes', 'necessary', 'for', 'an', 'elephant', 'to', 'cock', 'a', 'leg', 'to', 'allow', 'the', 'sun', 'to', 'go', 'past', 'Exactly', 'why', 'this', 'should', 'be', 'may', 'never', 'be', 'known', 'Possibly', 'the', 'Creator', 'of', 'the', 'universe', 'got', 'bored', 'with', 'all', 'the', 'usual', 'business', 'of', 'axial', 'inclination', 'albedos', 'and', 'rotational', 'velocities', 'and', 'decided', 'to', 'have', 'a', 'bit', 'of', 'fun', 'for', 'once', 'It', 'would', 'be', 'a', 'pretty', 'good', 'bet', 'that', 'the', 'gods', 'of', 'a', 'world', 'like', 'this', 'probably', 'do', 'not', 'play', 'chess', 'and', 'indeed', 'this', 'is', 'the', 'case', 'In', 'fact', 'no', 'gods', 'anywhere', 'play', 'chess', 'They', 'have', "n't", 'got', 'the', 'imagination', 'Gods', 'prefer', 'simple', 'vicious', 'games', 'where', 'you', 'Do', 'Not', 'Achieve', 'Transcendence', 'but', 'Go', 'Straight', 'To', 'Oblivion', 'a', 'key', 'to', 'the', 'understanding', 'of', 'all', 'religion', 'is', 'that', 'a', 'god', "'s", 'idea', 'of', 'amusement', 'is', 'Snakes', 'and', 'Ladders', 'with', 'greased', 'rungs', 'Magic', 'glues', 'the', 'Discworld', 'together', '–', 'magic', 'generated', 'by', 'the', 'turning', 'of', 'the', 'world', 'itself', 'magic', 'wound', 'like', 'silk', 'out', 'of', 'the', 'underlying', 'structure', 'of', 'existence', 'to', 'suture', 'the', 'wounds', 'of', 'reality', 'A', 'lot', 'of', 'it', 'ends', 'up', 'in', 'the', 'Ramtop', 'Mountains', 'which', 'stretch', 'from', 'the', 'frozen', 'lands', 'near', 'the', 'Hub', 'all', 'the', 'way', 'via', 'a', 'lengthy', 'archipelago', 'to', 'the', 'warm', 'seas', 'which', 'flow', 'endlessly', 'into', 'space', 'over', 'the', 'Rim', 'Raw', 'magic', 'crackles', 'invisibly', 'from', 'peak', 'to', 'peak', 'and', 'earths', 'itself', 'in', 'the', 'mountains', 'It', 'is', 'the', 'Ramtops', 'that', 'supply', 'the', 'world', 'with', 'most', 'of', 'its', 'witches', 'and', 'wizards', 'In', 'the', 'Ramtops', 'the', 'leaves', 'on', 'the', 'trees', 'move', 'even', 'when', 'there', 'is', 'no', 'breeze', 'Rocks', 'go', 'for', 'a', 'stroll', 'of', 'an', 'evening', 'Even', 'the', 'land', 'at', 'times', 'seems', 'alive', 'At', 'times', 'so', 'does', 'the', 'sky', 'The', 'storm', 'was', 'really', 'giving', 'it', 'everything', 'it', 'had', 'This', 'was', 'its', 'big', 'chance', 'It', 'had', 'spent', 'years', 'hanging', 'around', 'the', 'provinces', 'putting', 'in', 'some', 'useful', 'work', 'as', 'a', 'squall', 'building', 'up', 'experience', 'making', 'contacts', 'occasionally', 'leaping', 'out', 'on', 'unsuspecting', 'shepherds', 'or', 'blasting', 'quite', 'small', 'oak', 'trees', 'Now', 'an', 'opening', 'in', 'the', 'weather', 'had', 'given', 'it', 'an', 'opportunity', 'to', 'strut', 'its', 'hour', 'and', 'it', 'was', 'building', 'up', 'its', 'role', 'in', 'the', 'hope', 'of', 'being', 'spotted', 'by', 'one', 'of', 'the', 'big', 'climates', 'It', 'was', 'a', 'good', 'storm', 'There', 'was', 'quite', 'effective', 'projection', 'and', 'passion', 'there', 'and', 'critics', 'agreed', 'that', 'if', 'it', 'would', 'only', 'learn', 'to', 'control', 'its', 'thunder', 'it', 'would', 'be', 'in', 'years', 'to', 'come', 'a', 'storm', 'to', 'watch', 'The', 'woods', 'roared', 'their', 'applause', 'and', 'were', 'full', 'of', 'mists', 'and', 'flying', 'leaves']Your turn:
How many words are there in the first pdf page of Wyrd Sisters?
Sentences
Extracting sentences is just as easy with the sentences attribute.
Let’s extract the first 10 sentences:
text.sentences[:10][Sentence("
Terry Pratchett
Wyrd Sisters
(Starring Three Witches, also kings, daggers, crowns, storms, dwarfs, cats, ghosts, spectres,
apes, bandits, demons, forests, heirs, jesters, tortures, trolls, turntables, general rejoicing and
drivers alarums.)"),
Sentence("The wind howled."),
Sentence("Lightning stabbed at the earth erratically, like an inefficient assassin."),
Sentence("Thunder rolled back and forth across the dark, rain-lashed hills."),
Sentence("The night was as black as the inside of a cat."),
Sentence("It was the kind of night, you could believe, on
which gods moved men as though they were pawns on the chessboard of fate."),
Sentence("In the middle of this
elemental storm a fire gleamed among the dripping furze bushes like the madness in a weasel's eye."),
Sentence("It illuminated three hunched figures."),
Sentence("As the cauldron bubbled an eldritch voice shrieked: 'When
shall we three meet again?'"),
Sentence("There was a pause.")]The output is however quite ugly. We could make this a lot more readable by printing each sentence separated by a blank line:
for s in text.sentences[:10]:
print(s)
print("\n")
Terry Pratchett
Wyrd Sisters
(Starring Three Witches, also kings, daggers, crowns, storms, dwarfs, cats, ghosts, spectres,
apes, bandits, demons, forests, heirs, jesters, tortures, trolls, turntables, general rejoicing and
drivers alarums.)
The wind howled.
Lightning stabbed at the earth erratically, like an inefficient assassin.
Thunder rolled back and forth across the dark, rain-lashed hills.
The night was as black as the inside of a cat.
It was the kind of night, you could believe, on
which gods moved men as though they were pawns on the chessboard of fate.
In the middle of this
elemental storm a fire gleamed among the dripping furze bushes like the madness in a weasel's eye.
It illuminated three hunched figures.
As the cauldron bubbled an eldritch voice shrieked: 'When
shall we three meet again?'
There was a pause.
In Python strings (as in many other languages), "\n" represents a new line.
Or you could add lines of hyphens between the sentences:
for s in text.sentences[:10]:
print(s)
print("-" * 100)
Terry Pratchett
Wyrd Sisters
(Starring Three Witches, also kings, daggers, crowns, storms, dwarfs, cats, ghosts, spectres,
apes, bandits, demons, forests, heirs, jesters, tortures, trolls, turntables, general rejoicing and
drivers alarums.)
----------------------------------------------------------------------------------------------------
The wind howled.
----------------------------------------------------------------------------------------------------
Lightning stabbed at the earth erratically, like an inefficient assassin.
----------------------------------------------------------------------------------------------------
Thunder rolled back and forth across the dark, rain-lashed hills.
----------------------------------------------------------------------------------------------------
The night was as black as the inside of a cat.
----------------------------------------------------------------------------------------------------
It was the kind of night, you could believe, on
which gods moved men as though they were pawns on the chessboard of fate.
----------------------------------------------------------------------------------------------------
In the middle of this
elemental storm a fire gleamed among the dripping furze bushes like the madness in a weasel's eye.
----------------------------------------------------------------------------------------------------
It illuminated three hunched figures.
----------------------------------------------------------------------------------------------------
As the cauldron bubbled an eldritch voice shrieked: 'When
shall we three meet again?'
----------------------------------------------------------------------------------------------------
There was a pause.
----------------------------------------------------------------------------------------------------Your turn:
- What is the type of
text.sentences?
- Could you print just the 5th sentence?
- Just the last sentence?
Word counts
We already saw that we can extract words with the words attribute. Now, we can add the count method to get the frequency of specific words.
text.words.count("gods")4