A Gentle Introduction to Natural Language Process#
Today, we will take a text, War and Peace by Leo Tolstoy, and try to get verify or refute the claims of an outside source, Tolstoy’s Phoenix: From Method to Meaning in War and Peace by George R. Clay (1998). This is a very common task both in and outside of the digital humanities and will introduce you to the popular NLP Python package, the Natural Language Toolkit (NLTK), and expose you to common methodologies for wrangling textual data.
Our overall task#
We will start by downloading the book and then we will learn how to clean the text, perform basic statistics, create visualizations, and discuss what we found and how to present those results
Goals#
Understand the rights we have to access books on Project Gutenberg
Read in a text from Project Gutenberg
Clean textual data using regular expressions
Perform basic word frequency statistics
Create visualziations of these statistics
Discuss how to communicte thses results
Return to our research question
General methods#
Read in our data#
We will start with a url from Project Gutenberg. All of the texts from Gutenberg are in the public domain, so we won’t have to worry about rights, but be aware of who own the intellectual property to a text before you scrape it. In this section, we will break the text up by chapter division. Later we’ll do the same but by book division.
The requests library#
Here, we are using a library called requests. This library is great for HTTP requests, which are like asking for a specific action from the internet.
More details below: https://pypi.org/project/requests/
import requests
url = 'https://www.gutenberg.org/cache/epub/2600/pg2600.txt'
text = requests.get(url).text
## IMPORTANT: this pipeline will work for all text, but you will need to understand your text
## There is no one-size-fits-all for text cleaning, especially for extra information, like tables
## of content and disclaimers
# In this case, we found the first line that states the start of the book. We found this by looking at the actual book.
# Link to book: https://www.gutenberg.org/cache/epub/2600/pg2600.txt
start = text.find('BOOK ONE: 1805\r\n\r\n\r\n\r\n\r\n\r\n')
start
8265
end = text.find('END OF THE PROJECT GUTENBERG EBOOK WAR AND PEACE')
end
3274788
## Using the start and end points that we found above, we can filter out all of the text before the start of the book
## and after its end.
## In Python, we can use square brackets to delimit the new start and new end that we want. As we see below:
text = text[start:end]
Now that we have a text that we are confident is nothing but the text of the book, we can begin to dissect it into its component parts. First, let’s break it up into chapters.
To do so, I am using a very versatile package called re or regular expressions (regex) to do some advanced string parsing. This regex function, finditer, takes in a pattern and a text and returns all of the times that pattern occurs in the text. These patterns can look very complicated, but, in this case, it is ‘(Chapter [A-Z]+)’, which means: find all of the times when there is the word ‘CHAPTER’ followed by a space and then any amount of captial letters.
This pattern fits the roman numeral counting of the chapters (ex. CHAPTER I or CHAPTER XII)
Finditer also return the indices where the chapter title begins and ends. So, we can use the ending of one chapter title and the beginning of the next one to get all of the text in between the two chapters. This text is, by definition, the text in that chapter.
You can read more about regex here and you can play around with your own regex at regex101.
import re
ch_dict = {}
ch_list = list(re.finditer(r'(CHAPTER [A-Z]+)',text))
for i, m in enumerate(ch_list):
if i < len(ch_list)-1:
ch_dict[f'Chapter {i+1}'] = (m.end(0),ch_list[i+1].start(0)-1)
else:
ch_dict[f'Chapter {i+1}'] = (m.end(0), len(text)-1)
ch_dict
{'Chapter 1': (35, 11716),
'Chapter 2': (11727, 19652),
'Chapter 3': (19664, 28409),
'Chapter 4': (28420, 36599),
'Chapter 5': (36609, 47877),
'Chapter 6': (47888, 55785),
'Chapter 7': (55797, 61618),
'Chapter 8': (61631, 68528),
'Chapter 9': (68539, 80778),
'Chapter 10': (80788, 90636),
'Chapter 11': (90647, 95709),
'Chapter 12': (95721, 103553),
'Chapter 13': (103566, 107749),
'Chapter 14': (107761, 116408),
'Chapter 15': (116419, 125129),
'Chapter 16': (125141, 135663),
'Chapter 17': (135676, 139826),
'Chapter 18': (139840, 153522),
'Chapter 19': (153534, 160235),
'Chapter 20': (160246, 172138),
'Chapter 21': (172150, 187834),
'Chapter 22': (187847, 198012),
'Chapter 23': (198026, 208352),
'Chapter 24': (208365, 217847),
'Chapter 25': (217859, 238302),
'Chapter 26': (238315, 249685),
'Chapter 27': (249699, 258953),
'Chapter 28': (258968, 277450),
'Chapter 29': (277460, 287532),
'Chapter 30': (287543, 303366),
'Chapter 31': (303378, 317732),
'Chapter 32': (317743, 335287),
'Chapter 33': (335297, 342076),
'Chapter 34': (342087, 347628),
'Chapter 35': (347640, 357850),
'Chapter 36': (357863, 376832),
'Chapter 37': (376843, 387567),
'Chapter 38': (387577, 399996),
'Chapter 39': (400007, 405255),
'Chapter 40': (405267, 416777),
'Chapter 41': (416790, 429464),
'Chapter 42': (429476, 436196),
'Chapter 43': (436207, 448576),
'Chapter 44': (448588, 453728),
'Chapter 45': (453741, 464522),
'Chapter 46': (464536, 474352),
'Chapter 47': (474364, 486140),
'Chapter 48': (486151, 499062),
'Chapter 49': (499074, 515816),
'Chapter 50': (515826, 535228),
'Chapter 51': (535239, 554482),
'Chapter 52': (554494, 572446),
'Chapter 53': (572457, 589592),
'Chapter 54': (589602, 601924),
'Chapter 55': (601935, 614563),
'Chapter 56': (614575, 633355),
'Chapter 57': (633368, 643577),
'Chapter 58': (643588, 657352),
'Chapter 59': (657362, 668384),
'Chapter 60': (668395, 677348),
'Chapter 61': (677360, 691174),
'Chapter 62': (691187, 703199),
'Chapter 63': (703211, 714894),
'Chapter 64': (714905, 727688),
'Chapter 65': (727700, 736157),
'Chapter 66': (736170, 747241),
'Chapter 67': (747255, 761993),
'Chapter 68': (762005, 771296),
'Chapter 69': (771306, 788002),
'Chapter 70': (788013, 801020),
'Chapter 71': (801032, 812316),
'Chapter 72': (812327, 823160),
'Chapter 73': (823170, 828195),
'Chapter 74': (828206, 838301),
'Chapter 75': (838313, 845370),
'Chapter 76': (845383, 854584),
'Chapter 77': (854595, 859683),
'Chapter 78': (859693, 867016),
'Chapter 79': (867027, 872301),
'Chapter 80': (872313, 879240),
'Chapter 81': (879253, 885833),
'Chapter 82': (885845, 891870),
'Chapter 83': (891881, 899229),
'Chapter 84': (899241, 906388),
'Chapter 85': (906398, 913648),
'Chapter 86': (913659, 926615),
'Chapter 87': (926627, 941734),
'Chapter 88': (941745, 949937),
'Chapter 89': (949947, 953867),
'Chapter 90': (953878, 963628),
'Chapter 91': (963640, 966960),
'Chapter 92': (966973, 976016),
'Chapter 93': (976027, 987789),
'Chapter 94': (987799, 998043),
'Chapter 95': (998054, 1015627),
'Chapter 96': (1015639, 1023407),
'Chapter 97': (1023420, 1031896),
'Chapter 98': (1031908, 1036698),
'Chapter 99': (1036709, 1046151),
'Chapter 100': (1046163, 1057473),
'Chapter 101': (1057486, 1065212),
'Chapter 102': (1065226, 1071207),
'Chapter 103': (1071219, 1080110),
'Chapter 104': (1080121, 1088277),
'Chapter 105': (1088289, 1098614),
'Chapter 106': (1098627, 1099622),
'Chapter 107': (1099632, 1105115),
'Chapter 108': (1105126, 1110849),
'Chapter 109': (1110861, 1116058),
'Chapter 110': (1116069, 1122794),
'Chapter 111': (1122804, 1134448),
'Chapter 112': (1134459, 1141243),
'Chapter 113': (1141255, 1151007),
'Chapter 114': (1151020, 1157898),
'Chapter 115': (1157909, 1163462),
'Chapter 116': (1163472, 1174306),
'Chapter 117': (1174317, 1182434),
'Chapter 118': (1182446, 1187852),
'Chapter 119': (1187865, 1195693),
'Chapter 120': (1195705, 1203595),
'Chapter 121': (1203606, 1209863),
'Chapter 122': (1209875, 1218066),
'Chapter 123': (1218079, 1222459),
'Chapter 124': (1222473, 1232788),
'Chapter 125': (1232800, 1236768),
'Chapter 126': (1236779, 1243765),
'Chapter 127': (1243777, 1250307),
'Chapter 128': (1250320, 1258390),
'Chapter 129': (1258404, 1270856),
'Chapter 130': (1270869, 1277248),
'Chapter 131': (1277260, 1285069),
'Chapter 132': (1285082, 1292585),
'Chapter 133': (1292595, 1302392),
'Chapter 134': (1302403, 1306648),
'Chapter 135': (1306660, 1312807),
'Chapter 136': (1312818, 1325196),
'Chapter 137': (1325206, 1335367),
'Chapter 138': (1335378, 1350083),
'Chapter 139': (1350095, 1368974),
'Chapter 140': (1368987, 1375976),
'Chapter 141': (1375987, 1384187),
'Chapter 142': (1384197, 1402209),
'Chapter 143': (1402220, 1410852),
'Chapter 144': (1410864, 1417337),
'Chapter 145': (1417350, 1424143),
'Chapter 146': (1424153, 1435423),
'Chapter 147': (1435434, 1443815),
'Chapter 148': (1443827, 1456028),
'Chapter 149': (1456039, 1461442),
'Chapter 150': (1461452, 1471642),
'Chapter 151': (1471653, 1478639),
'Chapter 152': (1478651, 1486770),
'Chapter 153': (1486783, 1495198),
'Chapter 154': (1495209, 1506519),
'Chapter 155': (1506529, 1513793),
'Chapter 156': (1513804, 1519102),
'Chapter 157': (1519114, 1526244),
'Chapter 158': (1526257, 1532999),
'Chapter 159': (1533011, 1539464),
'Chapter 160': (1539475, 1550543),
'Chapter 161': (1550555, 1561717),
'Chapter 162': (1561730, 1566915),
'Chapter 163': (1566929, 1573960),
'Chapter 164': (1573972, 1581957),
'Chapter 165': (1581968, 1588633),
'Chapter 166': (1588645, 1597177),
'Chapter 167': (1597190, 1603512),
'Chapter 168': (1603522, 1613939),
'Chapter 169': (1613950, 1622742),
'Chapter 170': (1622754, 1631118),
'Chapter 171': (1631129, 1640271),
'Chapter 172': (1640281, 1645270),
'Chapter 173': (1645281, 1661005),
'Chapter 174': (1661017, 1668134),
'Chapter 175': (1668147, 1681915),
'Chapter 176': (1681926, 1698654),
'Chapter 177': (1698664, 1706589),
'Chapter 178': (1706600, 1718176),
'Chapter 179': (1718188, 1727716),
'Chapter 180': (1727729, 1734132),
'Chapter 181': (1734144, 1741147),
'Chapter 182': (1741158, 1748429),
'Chapter 183': (1748441, 1755235),
'Chapter 184': (1755248, 1763140),
'Chapter 185': (1763154, 1775008),
'Chapter 186': (1775020, 1783412),
'Chapter 187': (1783423, 1796121),
'Chapter 188': (1796133, 1808030),
'Chapter 189': (1808043, 1820752),
'Chapter 190': (1820766, 1824958),
'Chapter 191': (1824968, 1837385),
'Chapter 192': (1837396, 1846970),
'Chapter 193': (1846982, 1852552),
'Chapter 194': (1852563, 1875958),
'Chapter 195': (1875968, 1891721),
'Chapter 196': (1891732, 1901053),
'Chapter 197': (1901065, 1908516),
'Chapter 198': (1908529, 1927307),
'Chapter 199': (1927318, 1938576),
'Chapter 200': (1938586, 1949815),
'Chapter 201': (1949826, 1955504),
'Chapter 202': (1955516, 1960165),
'Chapter 203': (1960178, 1968513),
'Chapter 204': (1968525, 1979128),
'Chapter 205': (1979139, 1993246),
'Chapter 206': (1993258, 2000801),
'Chapter 207': (2000814, 2010415),
'Chapter 208': (2010429, 2021112),
'Chapter 209': (2021124, 2032585),
'Chapter 210': (2032596, 2040742),
'Chapter 211': (2040754, 2050948),
'Chapter 212': (2050961, 2058827),
'Chapter 213': (2058841, 2062981),
'Chapter 214': (2062994, 2069063),
'Chapter 215': (2069075, 2086019),
'Chapter 216': (2086032, 2094717),
'Chapter 217': (2094731, 2102215),
'Chapter 218': (2102230, 2108511),
'Chapter 219': (2108524, 2114563),
'Chapter 220': (2114575, 2121489),
'Chapter 221': (2121502, 2138539),
'Chapter 222': (2138553, 2142699),
'Chapter 223': (2142714, 2150076),
'Chapter 224': (2150090, 2160598),
'Chapter 225': (2160611, 2169695),
'Chapter 226': (2169709, 2181032),
'Chapter 227': (2181047, 2187682),
'Chapter 228': (2187698, 2195247),
'Chapter 229': (2195261, 2201533),
'Chapter 230': (2201543, 2209307),
'Chapter 231': (2209318, 2216996),
'Chapter 232': (2217008, 2223189),
'Chapter 233': (2223200, 2231358),
'Chapter 234': (2231368, 2237418),
'Chapter 235': (2237429, 2245212),
'Chapter 236': (2245224, 2254116),
'Chapter 237': (2254129, 2258748),
'Chapter 238': (2258759, 2265657),
'Chapter 239': (2265667, 2273244),
'Chapter 240': (2273255, 2278030),
'Chapter 241': (2278042, 2286957),
'Chapter 242': (2286970, 2294811),
'Chapter 243': (2294823, 2301093),
'Chapter 244': (2301104, 2308533),
'Chapter 245': (2308545, 2319288),
'Chapter 246': (2319301, 2329361),
'Chapter 247': (2329375, 2336641),
'Chapter 248': (2336653, 2346282),
'Chapter 249': (2346293, 2351634),
'Chapter 250': (2351646, 2357547),
'Chapter 251': (2357560, 2362664),
'Chapter 252': (2362678, 2371927),
'Chapter 253': (2371940, 2380747),
'Chapter 254': (2380759, 2401423),
'Chapter 255': (2401436, 2413736),
'Chapter 256': (2413750, 2422780),
'Chapter 257': (2422795, 2429139),
'Chapter 258': (2429152, 2451042),
'Chapter 259': (2451054, 2454769),
'Chapter 260': (2454782, 2464557),
'Chapter 261': (2464571, 2478191),
'Chapter 262': (2478206, 2492662),
'Chapter 263': (2492676, 2502383),
'Chapter 264': (2502393, 2510871),
'Chapter 265': (2510882, 2516536),
'Chapter 266': (2516548, 2522656),
'Chapter 267': (2522667, 2534007),
'Chapter 268': (2534017, 2541217),
'Chapter 269': (2541228, 2550621),
'Chapter 270': (2550633, 2560555),
'Chapter 271': (2560568, 2569428),
'Chapter 272': (2569439, 2574361),
'Chapter 273': (2574371, 2582900),
'Chapter 274': (2582911, 2591361),
'Chapter 275': (2591373, 2603577),
'Chapter 276': (2603590, 2610037),
'Chapter 277': (2610049, 2621780),
'Chapter 278': (2621791, 2630559),
'Chapter 279': (2630571, 2642981),
'Chapter 280': (2642991, 2650259),
'Chapter 281': (2650270, 2655080),
'Chapter 282': (2655092, 2661472),
'Chapter 283': (2661483, 2665803),
'Chapter 284': (2665813, 2669053),
'Chapter 285': (2669064, 2676866),
'Chapter 286': (2676878, 2681741),
'Chapter 287': (2681754, 2686389),
'Chapter 288': (2686400, 2694508),
'Chapter 289': (2694518, 2702171),
'Chapter 290': (2702182, 2711476),
'Chapter 291': (2711488, 2717734),
'Chapter 292': (2717747, 2725063),
'Chapter 293': (2725075, 2735213),
'Chapter 294': (2735224, 2740707),
'Chapter 295': (2740719, 2746541),
'Chapter 296': (2746554, 2753110),
'Chapter 297': (2753124, 2757166),
'Chapter 298': (2757178, 2761937),
'Chapter 299': (2761947, 2768946),
'Chapter 300': (2768957, 2774372),
'Chapter 301': (2774384, 2780726),
'Chapter 302': (2780737, 2788496),
'Chapter 303': (2788506, 2796405),
'Chapter 304': (2796416, 2801417),
'Chapter 305': (2801429, 2809444),
'Chapter 306': (2809457, 2814587),
'Chapter 307': (2814598, 2821733),
'Chapter 308': (2821743, 2830641),
'Chapter 309': (2830652, 2837084),
'Chapter 310': (2837096, 2844273),
'Chapter 311': (2844286, 2851082),
'Chapter 312': (2851094, 2854450),
'Chapter 313': (2854461, 2859825),
'Chapter 314': (2859837, 2863545),
'Chapter 315': (2863558, 2867438),
'Chapter 316': (2867452, 2871291),
'Chapter 317': (2871303, 2881724),
'Chapter 318': (2881734, 2890648),
'Chapter 319': (2890659, 2895611),
'Chapter 320': (2895623, 2901600),
'Chapter 321': (2901611, 2908666),
'Chapter 322': (2908676, 2916155),
'Chapter 323': (2916166, 2922637),
'Chapter 324': (2922649, 2927977),
'Chapter 325': (2927990, 2936152),
'Chapter 326': (2936163, 2941344),
'Chapter 327': (2941354, 2953701),
'Chapter 328': (2953712, 2957335),
'Chapter 329': (2957347, 2963825),
'Chapter 330': (2963838, 2975628),
'Chapter 331': (2975640, 2980768),
'Chapter 332': (2980779, 2987775),
'Chapter 333': (2987787, 2993311),
'Chapter 334': (2993324, 3004375),
'Chapter 335': (3004389, 3015383),
'Chapter 336': (3015395, 3019531),
'Chapter 337': (3019542, 3022806),
'Chapter 338': (3022816, 3029539),
'Chapter 339': (3029550, 3033720),
'Chapter 340': (3033732, 3043990),
'Chapter 341': (3044001, 3049520),
'Chapter 342': (3049530, 3056726),
'Chapter 343': (3056737, 3065865),
'Chapter 344': (3065877, 3073225),
'Chapter 345': (3073238, 3081182),
'Chapter 346': (3081193, 3093226),
'Chapter 347': (3093236, 3104287),
'Chapter 348': (3104298, 3111479),
'Chapter 349': (3111491, 3121881),
'Chapter 350': (3121894, 3129357),
'Chapter 351': (3129369, 3140886),
'Chapter 352': (3140897, 3151252),
'Chapter 353': (3151264, 3162117),
'Chapter 354': (3162127, 3172699),
'Chapter 355': (3172710, 3182727),
'Chapter 356': (3182739, 3187844),
'Chapter 357': (3187855, 3202014),
'Chapter 358': (3202024, 3208550),
'Chapter 359': (3208561, 3216644),
'Chapter 360': (3216656, 3224192),
'Chapter 361': (3224205, 3234131),
'Chapter 362': (3234142, 3246514),
'Chapter 363': (3246524, 3257927),
'Chapter 364': (3257938, 3261527),
'Chapter 365': (3261539, 3266522)}
Now we have all the text extracted! But… dictionaries are not the most useful data structure. They can be difficult to query and to get basic statisics from. So we will convert our dictionary into a more robust data structure, a dataframe from the pandas package.
import pandas as pd ## You will almost always see pandas imported this way.
## Let's see what happens when we input the dictionary directly
pd.DataFrame(ch_dict)
## it's close but not quite what we wanted
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 1
----> 1 import pandas as pd ## You will almost always see pandas imported this way.
2 ## Let's see what happens when we input the dictionary directly
3 pd.DataFrame(ch_dict)
ModuleNotFoundError: No module named 'pandas'
## pandas is programmed to look for numerical indices, which dictionaries (because they're an unordered data type) do not have
## we can coerse it though to accept string values as the index with the 'from_dict' method and the 'orient' keyword argument
## the 'reset_index' method will then turn our index into a column and give us an index for the rows
ch_df = pd.DataFrame.from_dict(ch_dict, orient='index')
ch_df = ch_df.reset_index()
ch_df = ch_df.rename(columns={'index':'chapter',0:'start_idx',1:'end_idx'})
ch_df
| chapter | start_idx | end_idx | |
|---|---|---|---|
| 0 | Chapter 1 | 35 | 11716 |
| 1 | Chapter 2 | 11727 | 19652 |
| 2 | Chapter 3 | 19664 | 28409 |
| 3 | Chapter 4 | 28420 | 36599 |
| 4 | Chapter 5 | 36609 | 47877 |
| ... | ... | ... | ... |
| 360 | Chapter 361 | 3224205 | 3234131 |
| 361 | Chapter 362 | 3234142 | 3246514 |
| 362 | Chapter 363 | 3246524 | 3257927 |
| 363 | Chapter 364 | 3257938 | 3261527 |
| 364 | Chapter 365 | 3261539 | 3266518 |
365 rows × 3 columns
Now that the dataframe is in the correct orientation, let’s use the indices that we got to select the texts of each chapter. All we need to do is input start_idx:end_idx into square brackets next to the text object. Below is an example.
s = ch_df.iloc[0]['start_idx'] ## .iloc[0] gives us the first element of the dataframe
e = ch_df.iloc[0]['end_idx']
text[s:e]
## the first full chapter of War and Peace
## you can change the number after iloc to change which chapter you see
'\r\n\r\n“Well, Prince, so Genoa and Lucca are now just family estates of the\r\nBuonapartes. But I warn you, if you don’t tell me that this means war,\r\nif you still try to defend the infamies and horrors perpetrated by that\r\nAntichrist—I really believe he is Antichrist—I will have nothing\r\nmore to do with you and you are no longer my friend, no longer my\r\n‘faithful slave,’ as you call yourself! But how do you do? I see I\r\nhave frightened you—sit down and tell me all the news.”\r\n\r\nIt was in July, 1805, and the speaker was the well-known Anna Pávlovna\r\nSchérer, maid of honor and favorite of the Empress Márya Fëdorovna.\r\nWith these words she greeted Prince Vasíli Kurágin, a man of high\r\nrank and importance, who was the first to arrive at her reception. Anna\r\nPávlovna had had a cough for some days. She was, as she said, suffering\r\nfrom la grippe; grippe being then a new word in St. Petersburg, used\r\nonly by the elite.\r\n\r\nAll her invitations without exception, written in French, and delivered\r\nby a scarlet-liveried footman that morning, ran as follows:\r\n\r\n“If you have nothing better to do, Count (or Prince), and if the\r\nprospect of spending an evening with a poor invalid is not too terrible,\r\nI shall be very charmed to see you tonight between 7 and 10—Annette\r\nSchérer.”\r\n\r\n“Heavens! what a virulent attack!” replied the prince, not in the\r\nleast disconcerted by this reception. He had just entered, wearing an\r\nembroidered court uniform, knee breeches, and shoes, and had stars on\r\nhis breast and a serene expression on his flat face. He spoke in that\r\nrefined French in which our grandfathers not only spoke but thought, and\r\nwith the gentle, patronizing intonation natural to a man of importance\r\nwho had grown old in society and at court. He went up to Anna Pávlovna,\r\nkissed her hand, presenting to her his bald, scented, and shining head,\r\nand complacently seated himself on the sofa.\r\n\r\n“First of all, dear friend, tell me how you are. Set your friend’s\r\nmind at rest,” said he without altering his tone, beneath the\r\npoliteness and affected sympathy of which indifference and even irony\r\ncould be discerned.\r\n\r\n“Can one be well while suffering morally? Can one be calm in times\r\nlike these if one has any feeling?” said Anna Pávlovna. “You are\r\nstaying the whole evening, I hope?”\r\n\r\n“And the fete at the English ambassador’s? Today is Wednesday. I\r\nmust put in an appearance there,” said the prince. “My daughter is\r\ncoming for me to take me there.”\r\n\r\n“I thought today’s fete had been canceled. I confess all these\r\nfestivities and fireworks are becoming wearisome.”\r\n\r\n“If they had known that you wished it, the entertainment would have\r\nbeen put off,” said the prince, who, like a wound-up clock, by force\r\nof habit said things he did not even wish to be believed.\r\n\r\n“Don’t tease! Well, and what has been decided about Novosíltsev’s\r\ndispatch? You know everything.”\r\n\r\n“What can one say about it?” replied the prince in a cold, listless\r\ntone. “What has been decided? They have decided that Buonaparte has\r\nburnt his boats, and I believe that we are ready to burn ours.”\r\n\r\nPrince Vasíli always spoke languidly, like an actor repeating a stale\r\npart. Anna Pávlovna Schérer on the contrary, despite her forty years,\r\noverflowed with animation and impulsiveness. To be an enthusiast had\r\nbecome her social vocation and, sometimes even when she did not\r\nfeel like it, she became enthusiastic in order not to disappoint the\r\nexpectations of those who knew her. The subdued smile which, though it\r\ndid not suit her faded features, always played round her lips expressed,\r\nas in a spoiled child, a continual consciousness of her charming defect,\r\nwhich she neither wished, nor could, nor considered it necessary, to\r\ncorrect.\r\n\r\nIn the midst of a conversation on political matters Anna Pávlovna burst\r\nout:\r\n\r\n“Oh, don’t speak to me of Austria. Perhaps I don’t understand\r\nthings, but Austria never has wished, and does not wish, for war. She\r\nis betraying us! Russia alone must save Europe. Our gracious sovereign\r\nrecognizes his high vocation and will be true to it. That is the one\r\nthing I have faith in! Our good and wonderful sovereign has to perform\r\nthe noblest role on earth, and he is so virtuous and noble that God will\r\nnot forsake him. He will fulfill his vocation and crush the hydra of\r\nrevolution, which has become more terrible than ever in the person of\r\nthis murderer and villain! We alone must avenge the blood of the just\r\none.... Whom, I ask you, can we rely on?... England with her commercial\r\nspirit will not and cannot understand the Emperor Alexander’s\r\nloftiness of soul. She has refused to evacuate Malta. She wanted to\r\nfind, and still seeks, some secret motive in our actions. What answer\r\ndid Novosíltsev get? None. The English have not understood and cannot\r\nunderstand the self-abnegation of our Emperor who wants nothing for\r\nhimself, but only desires the good of mankind. And what have they\r\npromised? Nothing! And what little they have promised they will not\r\nperform! Prussia has always declared that Buonaparte is invincible, and\r\nthat all Europe is powerless before him.... And I don’t believe a\r\nword that Hardenburg says, or Haugwitz either. This famous Prussian\r\nneutrality is just a trap. I have faith only in God and the lofty\r\ndestiny of our adored monarch. He will save Europe!”\r\n\r\nShe suddenly paused, smiling at her own impetuosity.\r\n\r\n“I think,” said the prince with a smile, “that if you had been\r\nsent instead of our dear Wintzingerode you would have captured the King\r\nof Prussia’s consent by assault. You are so eloquent. Will you give me\r\na cup of tea?”\r\n\r\n“In a moment. À propos,” she added, becoming calm again, “I am\r\nexpecting two very interesting men tonight, le Vicomte de Mortemart, who\r\nis connected with the Montmorencys through the Rohans, one of the best\r\nFrench families. He is one of the genuine émigrés, the good ones. And\r\nalso the Abbé Morio. Do you know that profound thinker? He has been\r\nreceived by the Emperor. Had you heard?”\r\n\r\n“I shall be delighted to meet them,” said the prince. “But\r\ntell me,” he added with studied carelessness as if it had only just\r\noccurred to him, though the question he was about to ask was the chief\r\nmotive of his visit, “is it true that the Dowager Empress wants\r\nBaron Funke to be appointed first secretary at Vienna? The baron by all\r\naccounts is a poor creature.”\r\n\r\nPrince Vasíli wished to obtain this post for his son, but others were\r\ntrying through the Dowager Empress Márya Fëdorovna to secure it for\r\nthe baron.\r\n\r\nAnna Pávlovna almost closed her eyes to indicate that neither she nor\r\nanyone else had a right to criticize what the Empress desired or was\r\npleased with.\r\n\r\n“Baron Funke has been recommended to the Dowager Empress by her\r\nsister,” was all she said, in a dry and mournful tone.\r\n\r\nAs she named the Empress, Anna Pávlovna’s face suddenly assumed an\r\nexpression of profound and sincere devotion and respect mingled with\r\nsadness, and this occurred every time she mentioned her illustrious\r\npatroness. She added that Her Majesty had deigned to show Baron Funke\r\nbeaucoup d’estime, and again her face clouded over with sadness.\r\n\r\nThe prince was silent and looked indifferent. But, with the womanly and\r\ncourtierlike quickness and tact habitual to her, Anna Pávlovna\r\nwished both to rebuke him (for daring to speak as he had done of a man\r\nrecommended to the Empress) and at the same time to console him, so she\r\nsaid:\r\n\r\n“Now about your family. Do you know that since your daughter came\r\nout everyone has been enraptured by her? They say she is amazingly\r\nbeautiful.”\r\n\r\nThe prince bowed to signify his respect and gratitude.\r\n\r\n“I often think,” she continued after a short pause, drawing nearer\r\nto the prince and smiling amiably at him as if to show that political\r\nand social topics were ended and the time had come for intimate\r\nconversation—“I often think how unfairly sometimes the joys of life\r\nare distributed. Why has fate given you two such splendid children?\r\nI don’t speak of Anatole, your youngest. I don’t like him,” she\r\nadded in a tone admitting of no rejoinder and raising her eyebrows.\r\n“Two such charming children. And really you appreciate them less than\r\nanyone, and so you don’t deserve to have them.”\r\n\r\nAnd she smiled her ecstatic smile.\r\n\r\n“I can’t help it,” said the prince. “Lavater would have said I\r\nlack the bump of paternity.”\r\n\r\n“Don’t joke; I mean to have a serious talk with you. Do you know\r\nI am dissatisfied with your younger son? Between ourselves” (and her\r\nface assumed its melancholy expression), “he was mentioned at Her\r\nMajesty’s and you were pitied....”\r\n\r\nThe prince answered nothing, but she looked at him significantly,\r\nawaiting a reply. He frowned.\r\n\r\n“What would you have me do?” he said at last. “You know I did all\r\na father could for their education, and they have both turned out fools.\r\nHippolyte is at least a quiet fool, but Anatole is an active one. That\r\nis the only difference between them.” He said this smiling in a way\r\nmore natural and animated than usual, so that the wrinkles round\r\nhis mouth very clearly revealed something unexpectedly coarse and\r\nunpleasant.\r\n\r\n“And why are children born to such men as you? If you were not a\r\nfather there would be nothing I could reproach you with,” said Anna\r\nPávlovna, looking up pensively.\r\n\r\n“I am your faithful slave and to you alone I can confess that my\r\nchildren are the bane of my life. It is the cross I have to bear. That\r\nis how I explain it to myself. It can’t be helped!”\r\n\r\nHe said no more, but expressed his resignation to cruel fate by a\r\ngesture. Anna Pávlovna meditated.\r\n\r\n“Have you never thought of marrying your prodigal son Anatole?” she\r\nasked. “They say old maids have a mania for matchmaking, and though I\r\ndon’t feel that weakness in myself as yet, I know a little person who\r\nis very unhappy with her father. She is a relation of yours, Princess\r\nMary Bolkónskaya.”\r\n\r\nPrince Vasíli did not reply, though, with the quickness of memory and\r\nperception befitting a man of the world, he indicated by a movement of\r\nthe head that he was considering this information.\r\n\r\n“Do you know,” he said at last, evidently unable to check the sad\r\ncurrent of his thoughts, “that Anatole is costing me forty thousand\r\nrubles a year? And,” he went on after a pause, “what will it be in\r\nfive years, if he goes on like this?” Presently he added: “That’s\r\nwhat we fathers have to put up with.... Is this princess of yours\r\nrich?”\r\n\r\n“Her father is very rich and stingy. He lives in the country. He is\r\nthe well-known Prince Bolkónski who had to retire from the army under\r\nthe late Emperor, and was nicknamed ‘the King of Prussia.’ He is\r\nvery clever but eccentric, and a bore. The poor girl is very unhappy.\r\nShe has a brother; I think you know him, he married Lise Meinen lately.\r\nHe is an aide-de-camp of Kutúzov’s and will be here tonight.”\r\n\r\n“Listen, dear Annette,” said the prince, suddenly taking Anna\r\nPávlovna’s hand and for some reason drawing it downwards. “Arrange\r\nthat affair for me and I shall always be your most devoted slave-slafe\r\nwith an f, as a village elder of mine writes in his reports. She is rich\r\nand of good family and that’s all I want.”\r\n\r\nAnd with the familiarity and easy grace peculiar to him, he raised the\r\nmaid of honor’s hand to his lips, kissed it, and swung it to and fro\r\nas he lay back in his armchair, looking in another direction.\r\n\r\n“Attendez,” said Anna Pávlovna, reflecting, “I’ll speak to\r\nLise, young Bolkónski’s wife, this very evening, and perhaps the\r\nthing can be arranged. It shall be on your family’s behalf that I’ll\r\nstart my apprenticeship as old maid.”\r\n\r\n\r\n\r\n\r\n\r'
This is great, but now how do we do it with the whole column?
We can use the .apply method from pandas, which allows us to cast a single function to every cell in a column. In this case, we need to use two different columns, so we will use a lambda expression. lambda expressions are a way to build very simple functions in Python. They are very flexible and support a lot of different features like flow control and regular expressions. Here is a useful starter guide to lambda from W3Schools.
For us, we need lambda expression that will pull a value for each column and return the text between those two values. Note the axis=1 keyword parameter. This tells pandas that we want to go through the columns not the rows.
ch_df['text'] = ch_df.apply(lambda x: text[x['start_idx']:x['end_idx']],axis=1)
ch_df['text'] = ch_df['text'].apply(lambda x: x.replace('\r','').replace('\n',' ').replace(' ', ' ').lower()) ## using lambda again for some simple cleaning with the replace and lower methods
ch_df = ch_df.drop(['start_idx','end_idx'],axis=1) ## and now we can drop our indices just so they don't clutter up our dataframe
ch_df
| chapter | text | |
|---|---|---|
| 0 | Chapter 1 | “well, prince, so genoa and lucca are now jus... |
| 1 | Chapter 2 | anna pávlovna’s drawing room was gradually fi... |
| 2 | Chapter 3 | anna pávlovna’s reception was in full swing. ... |
| 3 | Chapter 4 | just then another visitor entered the drawing... |
| 4 | Chapter 5 | “and what do you think of this latest comedy,... |
| ... | ... | ... |
| 360 | Chapter 361 | if history dealt only with external phenomena... |
| 361 | Chapter 362 | for the solution of the question of free will... |
| 362 | Chapter 363 | thus our conception of free will and inevitab... |
| 363 | Chapter 364 | history examines the manifestations of man’s ... |
| 364 | Chapter 365 | from the time the law of copernicus was disco... |
365 rows × 2 columns
Tokenization#
Now that we have read in our data and began some cleaning, we will tokenize our text. Tokenization is a process by which we can split up the chapter we got into relevent units. There are two main types of tokenization:
Sentence-level tokenization: splitting up the text into sentences
Word-level tokenization: splitting up the text into words
## The Natural Language Toolkit (NLTK) provides a lot of very useful utilities for cleaning and analyzing textual data.
## It requires this 'punkt' download for us to be able to tokenize
import nltk
nltk.download('punkt')
[nltk_data] Downloading package punkt to /Users/pnadel01/nltk_data...
[nltk_data] Package punkt is already up-to-date!
True
Sentence-level tokenization#
ch_df['sents'] = ch_df.text.apply(nltk.sent_tokenize)
type(ch_df['sents'].iloc[0]) ## each element of the 'sents' column is a list of sentences in eahc chapter
list
## Now we can 'explode' these lists, so that each sentence has its own cell in the 'sents' column.
sent_explode = ch_df.explode('sents').drop(['text'],axis=1).reset_index(drop=True)
sent_explode
| chapter | sents | |
|---|---|---|
| 0 | Chapter 1 | “well, prince, so genoa and lucca are now jus... |
| 1 | Chapter 1 | but i warn you, if you don’t tell me that this... |
| 2 | Chapter 1 | but how do you do? |
| 3 | Chapter 1 | i see i have frightened you—sit down and tell ... |
| 4 | Chapter 1 | with these words she greeted prince vasíli kur... |
| ... | ... | ... |
| 26256 | Chapter 365 | as in the question of astronomy then, so in th... |
| 26257 | Chapter 365 | in astronomy it was the immovability of the ea... |
| 26258 | Chapter 365 | as with astronomy the difficulty of recognizin... |
| 26259 | Chapter 365 | but as in astronomy the new view said: “it is ... |
| 26260 | Chapter 365 | *** |
26261 rows × 2 columns
Word-level tokenization#
To conduct word-level tokenization, we’ll be using regex again but this time, we’ll use a useful function in NLTK that takes in a regular expression and will split words on that pattern. This is opposition to so-called ‘white-space tokenization’ where words are delimited by spaces. This type of tokenization can be useful in some cases, but in most natural language will be insufficient. That’s why this NLTK function is so helpful. All we need to do is input the pattern below and we will get very good English tokenization
Note that tokenization is language dependent. This means that just because a certain method works for English it will not neccesarily work for other languages.
from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r"(?x)\w+(?:[-’]\w+)*")
ex_sentence = sent_explode.sents.iloc[5]
print(ex_sentence)
tokenizer.tokenize(ex_sentence)
anna pávlovna had had a cough for some days.
['anna', 'pávlovna', 'had', 'had', 'a', 'cough', 'for', 'some', 'days']
sent_explode['word_tokenized'] = sent_explode['sents'].apply(tokenizer.tokenize)
sent_explode
| chapter | sents | word_tokenized | |
|---|---|---|---|
| 0 | Chapter 1 | “well, prince, so genoa and lucca are now jus... | [well, prince, so, genoa, and, lucca, are, now... |
| 1 | Chapter 1 | but i warn you, if you don’t tell me that this... | [but, i, warn, you, if, you, don’t, tell, me, ... |
| 2 | Chapter 1 | but how do you do? | [but, how, do, you, do] |
| 3 | Chapter 1 | i see i have frightened you—sit down and tell ... | [i, see, i, have, frightened, you, sit, down, ... |
| 4 | Chapter 1 | with these words she greeted prince vasíli kur... | [with, these, words, she, greeted, prince, vas... |
| ... | ... | ... | ... |
| 26256 | Chapter 365 | as in the question of astronomy then, so in th... | [as, in, the, question, of, astronomy, then, s... |
| 26257 | Chapter 365 | in astronomy it was the immovability of the ea... | [in, astronomy, it, was, the, immovability, of... |
| 26258 | Chapter 365 | as with astronomy the difficulty of recognizin... | [as, with, astronomy, the, difficulty, of, rec... |
| 26259 | Chapter 365 | but as in astronomy the new view said: “it is ... | [but, as, in, astronomy, the, new, view, said,... |
| 26260 | Chapter 365 | *** | [] |
26261 rows × 3 columns
Visualization#
We can now create a general function to visualize word use over the course of the book. For this task, we are more interested in general trends, so let’s zoom back out to the book-level.
For more information on how to create visualization in Python, please go through this training.
## creating a new dataframe organized by book
## this is the same code as above
book_dict = {}
book_list = list(re.finditer(r'(BOOK [A-Z]+:)',text))
for i, m in enumerate(book_list):
if i < len(book_list)-1:
book_dict[f'Book {i+1}'] = (m.end(0),book_list[i+1].start(0)-1)
else:
book_dict[f'Book {i+1}'] = (m.end(0), len(text)-1)
book_df = pd.DataFrame.from_dict(book_dict, orient='index').reset_index().rename(columns={'index':'book',0:'start_idx',1:'end_idx'})
book_df['text'] = book_df.apply(lambda x: text[x['start_idx']:x['end_idx']],axis=1)
book_df['text'] = book_df['text'].apply(lambda x: x.replace('\r','').replace('\n',' ').replace(' ', ' ').lower())
book_df = book_df.drop(['start_idx','end_idx'],axis=1)
book_df['word_tokenized'] = book_df.text.apply(tokenizer.tokenize)
search = 'napoleon'
book_df['search_count'] = book_df['word_tokenized'].apply(lambda x: x.count(search.lower()))
book_df.plot(x='book',y='search_count', kind='bar')
<AxesSubplot: xlabel='book'>
Generalizing#
Once I am happy with how my pipeline is functioning, I like to then try to write a function that is as general as possible. In this case, we started with a Gutenberg URL to a txt file, so let’s take that as the starting place for our general function. Please take some time to develop a function which will return a DataFrame which we can use to generate a similar visualization as above from any Gutenbery URL. Be careful to note where, in the code above, some methods are specific to War and Peace and which are general.
def get_gutenberg_data(
url, # the Gutenberg URL
# you may need more arguments for your function
):
"""
Takes in a Gutenberg URL
Returns a DataFrame of relevant information
"""
## GET TEXT FROM URL WITH A REQUEST
## TRIM YOUR TEXT BASED ON SUBSTRINGS
content_dict = {}
content_list = list(re.finditer(r'YOUR REGEX HERE!!'), text)
## CREATE A DICTIONARY FOR THE INDEX POSITONS YOUR CONTENT
## CONVERT YOUR DICTIONARY INTO A DATAFRAME
## CREATE A TEXT COLUMN IN YOUR DATAFRAME
## CLEAN YOUR TEXT
## TOKENIZE (SENTENCE AND WORD LEVEL)
## RETURN YOUR DATAFRAME
search = '' ## INPUT SEARCH TERM HERE
## ADD A COUNTS COLUMN
## PLOT A GRAPH
Specific Application#
From Tolstoy’s Phoenix: From Method to Meaning in War and Peace by George R. Clay (1998), page 7:
In the most direct use of Tolstoy’s technqiues, he categorizes in his own voice. Instead of “The princess smiled, thinking she know more abou the subject than Prince Vasili,” he will write: “The princess smiled, as people do who think they know more about the subject under discussion than those they are talking with” (1:2, p.77; emphasis added). The first version expresses a private thought, but the one Tolstoy used implies (as R. F. Christian phrased it) that “there is a basic denominator of human experience” – a sameness about our pattern of behavior, so that we all know what kind of smile it is… By writing “as people do,” Tolstoy isn’t telling us what this smile is like, he is assuming that we know what it is like: that we have seen it many times before…
Let’s try to find more examples and see if they fall in line with Clay’s analysis.
# the phrases Clay identifies as relevent
phrases = [
'as people do',
'of one who',
'peculiar to',
'as with everyone',
'as one who',
'only used by persons who',
'of a man who',
'which a man has',
'in the way',
'as is usually the case',
'such as often occurs',
'which usually follows'
]
def printExcerpt(row):
print(row['chapter'], row['sents'], sep='\t')
print('\n')
for phrase in phrases:
print(phrase.upper())
sub_df = sent_explode.loc[sent_explode['sents'].str.contains(phrase)]
sub_df.apply(printExcerpt,axis=1)
print('-----')
AS PEOPLE DO
Chapter 21 do you understand that in consideration of the count’s services, his request would be granted?...” the princess smiled as people do who think they know more about the subject under discussion than those they are talking with.
-----
OF ONE WHO
Chapter 6 he put on the air of one who finds it impossible to reply to such nonsense, but it would in fact have been difficult to give any other answer than the one prince andrew gave to this naïve question.
Chapter 10 “i can just imagine what a funny figure that policeman cut!” and as he waved his arms to impersonate the policeman, his portly form again shook with a deep ringing laugh, the laugh of one who always eats well and, in particular, drinks well.
Chapter 21 she had the air of one who has suddenly lost faith in the whole human race.
Chapter 113 when he had joined the freemasons he had experienced the feeling of one who confidently steps onto the smooth surface of a bog.
Chapter 150 but in spite of that she seemed to be disillusioned about everything and told everyone that she did not believe either in friendship or in love, or any of the joys of life, and expected peace only “yonder.” she adopted the tone of one who has suffered a great disappointment, like a girl who has either lost the man she loved or been cruelly deceived by him.
Chapter 170 the emperor, with the agitation of one who has been personally affronted, was finishing with these words: “to enter russia without declaring war!
-----
PECULIAR TO
Chapter 1 she is rich and of good family and that’s all i want.” and with the familiarity and easy grace peculiar to him, he raised the maid of honor’s hand to his lips, kissed it, and swung it to and fro as he lay back in his armchair, looking in another direction.
Chapter 51 prince vasíli frowned, twisting his mouth, his cheeks quivered and his face assumed the coarse, unpleasant expression peculiar to him.
Chapter 75 i will follow.” when princess mary returned from her father, the little princess sat working and looked up with that curious expression of inner, happy calm peculiar to pregnant women.
Chapter 150 julie on the contrary accepted his attentions readily, though in a manner peculiar to herself.
-----
AS WITH EVERYONE
Chapter 18 berg, oblivious of irony or indifference, continued to explain how by exchanging into the guards he had already gained a step on his old comrades of the cadet corps; how in wartime the company commander might get killed and he, as senior in the company, might easily succeed to the post; how popular he was with everyone in the regiment, and how satisfied his father was with him.
Chapter 25 as with everyone, her face assumed a forced unnatural expression as soon as she looked in a glass.
-----
AS ONE WHO
Chapter 8 halfway through supper prince andrew leaned his elbows on the table and, with a look of nervous agitation such as pierre had never before seen on his face, began to talk—as one who has long had something on his mind and suddenly determines to speak out.
-----
ONLY USED BY PERSONS WHO
Chapter 17 hey, who’s there?” he called out in a tone only used by persons who are certain that those they call will rush to obey the summons.
-----
OF A MAN WHO
Chapter 5 “if buonaparte remains on the throne of france a year longer,” the vicomte continued, with the air of a man who, in a matter with which he is better acquainted than anyone else, does not listen to others but follows the current of his own thoughts, “things will have gone too far.
Chapter 10 as soon as he had seen a visitor off he returned to one of those who were still in the drawing room, drew a chair toward him or her, and jauntily spreading out his legs and putting his hands on his knees with the air of a man who enjoys life and knows how to live, he swayed to and fro with dignity, offered surmises about the weather, or touched on questions of health, sometimes in russian and sometimes in very bad but self-confident french; then again, like a man weary but unflinching in the fulfillment of duty, he rose to see some visitors off and, stroking his scanty gray hairs over his bald patch, also asked them to dinner.
Chapter 37 reviewing his impressions of the recent battle, picturing pleasantly to himself the impression his news of a victory would create, or recalling the send-off given him by the commander in chief and his fellow officers, prince andrew was galloping along in a post chaise enjoying the feelings of a man who has at length begun to attain a long-desired happiness.
Chapter 37 his face took on the stupid artificial smile (which does not even attempt to hide its artificiality) of a man who is continually receiving many petitioners one after another.
Chapter 43 “now what does this mean, gentlemen?” said the staff officer, in the reproachful tone of a man who has repeated the same thing more than once.
Chapter 46 it expressed the concentrated and happy resolution you see on the face of a man who on a hot day takes a final run before plunging into the water.
Chapter 72 he was entirely absorbed by two considerations: his wife’s guilt, of which after his sleepless night he had not the slightest doubt, and the guiltlessness of dólokhov, who had no reason to preserve the honor of a man who was nothing to him.... “i should perhaps have done the same thing in his place,” thought pierre.
Chapter 177 he said a few words to prince andrew and chernýshev about the present war, with the air of a man who knows beforehand that all will go wrong, and who is not displeased that it should be so.
Chapter 258 i must tell you, mon cher,” he continued in the sad and measured tones of a man who intends to tell a long story, “that our name is one of the most ancient in france.” and with a frenchman’s easy and naïve frankness the captain told pierre the story of his ancestors, his childhood, youth, and manhood, and all about his relations and his financial and family affairs, “ma pauvre mère” playing of course an important part in the story.
Chapter 358 the replies this theory gives to historical questions are like the replies of a man who, watching the movements of a herd of cattle and paying no attention to the varying quality of the pasturage in different parts of the field, or to the driving of the herdsman, should attribute the direction the herd takes to what animal happens to be at its head.
Chapter 363 so that if we examine the case of a man whose connection with the external world is well known, where the time between the action and its examination is great, and where the causes of the action are most accessible, we get the conception of a maximum of inevitability and a minimum of free will.
-----
WHICH A MAN HAS
Chapter 173 napoleon was in that state of irritability in which a man has to talk, talk, and talk, merely to convince himself that he is in the right.
Chapter 334 now that he was telling it all to natásha he experienced that pleasure which a man has when women listen to him—not clever women who when listening either try to remember what they hear to enrich their minds and when opportunity offers to retell it, or who wish to adopt it to some thought of their own and promptly contribute their own clever comments prepared in their little mental workshop—but the pleasure given by real women gifted with a capacity to select and absorb the very best a man shows of himself.
-----
IN THE WAY
Chapter 18 he was in the way and was the only one who did not notice the fact.
Chapter 56 berg put on the cleanest of coats, without a spot or speck of dust, stood before a looking glass and brushed the hair on his temples upwards, in the way affected by the emperor alexander, and, having assured himself from the way rostóv looked at it that his coat had been noticed, left the room with a pleasant smile.
Chapter 128 sónya was afraid to leave natásha and afraid of being in the way when she was with them.
Chapter 130 the old countess sighed as she looked at them; sónya was always getting frightened lest she should be in the way and tried to find excuses for leaving them alone, even when they did not wish it.
Chapter 152 she knew what she ought to have said to natásha, but she had been unable to say it because mademoiselle bourienne was in the way, and because, without knowing why, she felt it very difficult to speak of the marriage.
Chapter 208 the stout man rose, frowned, shrugged his shoulders, and evidently trying to appear firm began to pull on his jacket without looking about him, but suddenly his lips trembled and he began to cry, in the way full-blooded grown-up men cry, though angry with himself for doing so.
Chapter 221 once or twice he was shouted at for being in the way.
Chapter 234 those who went away, taking what they could and abandoning their houses and half their belongings, did so from the latent patriotism which expresses itself not by phrases or by giving one’s children to save the fatherland and similar unnatural exploits, but unobtrusively, simply, organically, and therefore in the way that always produces the most powerful results.
Chapter 282 in all these plottings the subject of intrigue was generally the conduct of the war, which all these men believed they were directing; but this affair of the war went on independently of them, as it had to go: that is, never in the way people devised, but flowing always from the essential attitude of the masses.
-----
AS IS USUALLY THE CASE
Chapter 95 as is usually the case with people meeting after a prolonged separation, it was long before their conversation could settle on anything.
-----
SUCH AS OFTEN OCCURS
Chapter 139 after a casual pause, such as often occurs when receiving friends for the first time in one’s own house, “uncle,” answering a thought that was in his visitors’ minds, said: “this, you see, is how i am finishing my days... death will come.
-----
WHICH USUALLY FOLLOWS
Chapter 334 they all three of them now experienced that feeling of awkwardness which usually follows after a serious and heartfelt talk.
-----
Great! Now we can see all of the times that Tolstoy uses these particular phrases and we can begin to see what Clay is saying. In this lines, Tolstoy is not giving us a direct characterization or description of a person or thing. Instead he is telling us about the type of person or thing it is. The next step would be then to see if there is a trend in the people or things he refers to in this manner or if this is, as Clay contends, is a general feature of Tolstoy’s prose. At every step in that process, we must make sure to compare our results with Clay and Tolstoy himself. It can be easy to trick yourself into a discovery and the only way to avoid that is to have a close connection to the data itself. Always spend time to play around with your data, even after you think it’s 100% clean, to see what it hides.
Reviewing what we learned#
How to take advantage of free repositories of text like Project Gutenberg
Taking a url to a text and reading it into our Python runtime
Cleaning our data so that we can find some general statisics
How to create visualizations of these statisics
Applying simple methodologies to a complex question
Outlining a method we could use to answer such a question
As a challenge, now that we have a good baseline for picking out the phrases Clay points us too, can you create dataframe that stores the sentence, what phrase from Clay it uses, and what chapter it is in? Be sure to start with a dictionary, as we did when we created a dataframe together and then convert it into a dataframe using from_dict. When you’re done, you can then export it as a .csv file, so you can use it later with the method whatever_your_df_is_called.to_csv('name_of_file.csv'). This way you can share your work with other scholars and later can deploy it to a website or another publication. If you have trouble or just want to show off how you did it, feel free to reach out and let me know at peter.nadel@tufts.edu