Named Entity Recognition (NER)

Comments

Use the Stanford CoreNLP with nltk

NER is basically extracting the most important keywords in the article and categorizing it according to the Table below.

NE Type Example
Organization Microsoft, Google
Person Jack Ma, Bill Gates
Location Mount Everest, Amazon River
Date June, 2012-05-12
Time three fifty pm, 2:30pm
Money 200 million pounds, AUD 20.50
Percent twenty pct, 20.5%
Facility Uluru, Sydney Harbor bridge
GPE (Geo-Political Entities) South America, Middle East

Import modules

In [152]:
import os
import nltk
import collections
import matplotlib.pyplot as plt
from collections import defaultdict
from pprint import pprint
from nltk.tokenize import word_tokenize
from nltk.tokenize import sent_tokenize

nltk.download('tagsets')  # for `nltk.help.upenn_tagset('xxx')`
nltk.download('averaged_perceptron_tagger') # for nltk.pos_tag
nltk.download('maxent_ne_chunker') # for nltk.ne_chunk_sents 
nltk.download('words') # for nltk.ne_chunk_sentsv

[nltk_data] Downloading package tagsets to
[nltk_data]     C:\Users\xxklow\AppData\Roaming\nltk_data...
[nltk_data]   Package tagsets is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     C:\Users\xxklow\AppData\Roaming\nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package maxent_ne_chunker to
[nltk_data]     C:\Users\xxklow\AppData\Roaming\nltk_data...
[nltk_data]   Package maxent_ne_chunker is already up-to-date!
[nltk_data] Downloading package words to
[nltk_data]     C:\Users\xxklow\AppData\Roaming\nltk_data...
[nltk_data]   Package words is already up-to-date!
Out[152]:
True

Reading a file

The below article was downloaded from aeon.co which is a great website that I read whenever I can! The article is entitled How materialism became an ethos of hope for Jewish reformers that I've used as an example in this exercise.

In [3]:
currDir = os.getcwd()
fileName = 'aeon.txt'

readFile = currDir + '\\inputs\\' + fileName

f = open(readFile,'r')
article = f.read()
f.close()
print(article)

Be ‘a man in the street and a Jew in the home’: a common piece of advice that liberal Jews often gave their co-religionists in the 19th century. If Jewishness was kept invisible and private, they wagered, then Jews could become citizens and professionals, and be granted equal access to the material resources made available to any other member of society. There was plenty of Christian bias to combat, encapsulated by images of Jewish avarice and materialism such as Shylock’s greedy hands and Rothschild’s beard in the form of snake-like tentacles. If only Jews could fit into the spiritual boxes established by the European Protestant elite, they would be accepted, or at least tolerated in the public sphere as Frenchmen, Germans or Englishmen. Though compelling in theory, the deal became more fraught as rampant anti-Semitic violence in eastern Europe continued to remind Jews that, no matter how much they tried to look like ‘everyone else’, their bodies were marked as Jewish. 
In the 1870s European Judaism underwent an intellectual revolution. Around then, a group of young Russian Jewish radicals began to identify Judaism with materialism, and to theorise about what they called – whether in Russian, German, Yiddish or Hebrew – the ‘material’ (material’nii, materiell, gashmi, ?omri) aspects of the Universe. For many Jews living in this period, ‘materialism’ was a worldview that brought into focus latent Jewish ideas and beliefs about the physical world. The materialists claimed that a theory of Judaism, defined by the way people related to land, labour and bodies, had been lying dormant within Jewish literature – in Hasidic texts, the Bible, Spinoza’s philosophy – and could now be clearly recognised and fully articulated. Jewish particularity was based on specific historical economic differences between Jews and others. What made Jews different was a certain socioeconomic dynamic that distinguished them from their neighbours.
The Jewish revolutionaries in 1870s Russia who embraced the idea of materialism shared a number of critical assumptions. They all rejected the notion that Judaism was based on abstract metaphysical theories (Scholasticism), rituals (Hasidism), study (Mitnagdim), and ethics and reason (Enlighteners). Judaism was not a religion, like Protestantism. Instead it was something attached to their bodies and expressed through one’s relationship to land, labour and resources. The materialists had also given up hope that the state could protect them and ensure their economic wellbeing. And finally, they no longer believed that history was headed in a positive direction. Over no amount of time would Jews living in Russia ever be granted greater rights and opportunities. Therefore, only a radical reclaiming of the physical world on the part of Jews could ensure that they would be protected and given a fair and equal share of resources.
Soon, the Jewish materialism of the Russians could be found among western European Jews residing in England and Germany. Only half-jokingly, the German anarchist Gustav Landauer claimed in 1921 that what distinguished ‘the modern “conscious” Jew from a German was that when the latter writes about … the conservation of energy, … he writes about the conservation of energy, but when the conscious Jew writes about the conservation of energy, he writes about the conservation of energy and Judaism’ (emphasis mine). Eventually, there would be those, such as the Englishman Israel Zangwill, who considered themselves adherents to ‘a religion of pots and pans’, and others who identified Judaism as a faith based on ‘bagels and lox’. Over the course of the 20th century, Jews would increasingly come to believe that ‘there is nothing purely spiritual that stands on its own … Everything spiritual requires a necessary material basis.’
Updates on everything new at Aeon.
Top of Form
Bottom of Form
JJjjjJJ  Jewish materialists were despised not only by staunch liberals but also by ‘defenders of the faith’. Moses Leib Lilienblum, who would go on to found the Zionist movement in Russia, wrote a novel in which he described his youthful yeshiva education as one long masturbatory experience – for this, he was denounced by rabbis and communal leaders who forced him to flee his hometown in fear for his life. The future Russian revolutionary Hasia Schur was pelted with stones and jeered at by the townspeople of Mohilev for going on a Sabbath walk hand-in-hand with her boyfriend, the socialist Eliezer Tsukerman: the rabbis were up in arms that two young people had dared to touch one another in public. Jewish materialists were cast as upstarts, deviants, social provocateurs and, of course, with providing Jew-haters with excuses to promote anti-Semitism.
But the Jewish materialists’ deviancies reflected a radically new kind of Jewish identity, one focused on their bodies and the physical world. The Jewish body they imagined would offer a contrast to both the hunchbacked, traditional Jewish Torah scholar incapable of supporting his family, and the muscular gentile male whose energies were directed at conquering and dominating the physical world. The new Jewish body would be shaped in the image of a healthy traditional Jewish woman who laboured to provide for her family’s material wellbeing while her husband spent his day in the house of study: by tending to the material aspects of existence, Jews’ needs and desires would now be seen as the primary feature of Judaism. The material Jewish identity set the stage for Jews’ involvement in 20th-century politics: Zionism, Bundism (the Jewish labour movement), the Minority Rights movement, and Jewish forms of communism all assumed that the organising structure of Jewish identity was a Jewish body, and not a Judaism of the heavens or the heart. Jewish materialism made Jews political without them possessing their own state or even citizenship in a host country.
Though the idea of the Jewish body as the locus of collective identity would always be suspect in western Europe, it would, however, become the basis of a new kind of Jewish identity most commonly witnessed in Israel and the United States. Jewish immigrants to Palestine at the turn of the century saw in Zion the actualisation of materialism as first imagined in the 1870s. The Marxist Ber Borchov’s students, such as future leaders of Israel Yitzhak Ben-Zvi and David Ben-Gurion, identified Palestine as a response to the crisis of the fork and the knife (a pithy phrase meant to capture the economic challenges of Russian Jews in the 1870s) originally theorised by the Jewish materialist Aaron Shemuel Lieberman in the 1870s. They envisioned a new kind of Jew – the ?aluts (pioneer) – who was attached to the physical world. As described by the 20th-century Zionist poet Avraham Shlonsky, a former Hasidic Jew, the ?aluts would be the embodiment of the idea that ‘a human being is meat, and he toils here in the sacred/and the land/bread’. The people of the book had now become a people of labour, land and the body.
In the US, eastern European Jews established large-scale defence organisations directed at protecting Jewish bodies and providing a platform for Jews to speak as a distinct ethnic minority in the American public sphere. From the poet Emma Lazarus to the American rabbi Mordecai Kaplan to the philosopher Horace Kallen, American Jews in the early 20th century developed political programmes and established organisations rooted around the physical aspects of Jewish life.
Jewish materialism remains the defining element of most American Jews’ identity. Following the Second World War, the influx of another wave of Jewish immigrants from Russian lands gave rise to a new brand of US literature that placed the Jewish body front and centre. The late US novelist Phillip Roth might have been familiar only in passing with the name Moses Lilienblum. But it was Lilienblum who put into circulation the Jewish genre of overbearing parents, unrealisable social expectations, failed sexual encounters, silly rabbis, bankrupt synagogues and God-fearing charlatans encased in a narrative about masturbation. Whether he knew it or not, when Roth wrote his novel Portnoy’s Complaint(1969), he was channelling the same tradition first articulated by Lilienblum a century earlier.
Roth took those commitments to his grave when he died on 22 May 2018. While the grandmaster of late-20th-century American letters asked to be interred next to Jews, he strictly prohibited the performance of any Jewish rituals at his funeral. His final requests, allegedly, were inspired by a desire ‘to have someone to talk to’. His corpse did not need a rabbi to eulogise it, or a perfunctory kaddish (or hymn) to kasher it; it was simply Jewish – nothing more and nothing less. Indeed, it was a fitting conclusion to the life of a Jewish materialist.

*** Tagsets

Code Description
NN noun, common, singular, or mass
RB adverbs
CC conjunction, co-ordinating
IN preposition, conjunction, subordinating
JJ adjective, or numeral, ordinal
VBP verb, present tense, not 3rd person singular
NNP noun, proper, singular

There are many tagsets out there. use the nltk.help.upenn_tagset('VBP') to obtain information on each tag.

In [109]:
tagSet = 'NN'
nltk.help.upenn_tagset(tagSet)

NN: noun, common, singular or mass
    common-carrier cabbage knuckle-duster Casino afghan shed thermostat
    investment slide humour falloff slick wind hyena override subhumanity
    machinist ...

Pos: Part-of-speech tagging

  1. We tokenize the article into sentences using sent_tokenize
  2. Using word_tokenize(sentence) will create a list of sentence lists. The sentence list comprises of tokens of each sentence.
  3. Using nltk.pos_tag, we POS-tag each token of the sentence using POS (parts-of-speech) tagging so we know what type of category each word is. This produce a list of tuples where (i) token (ii) the POS-tag
  4. Using nltk.ne_chunk_sents) on the list of tuples consiting of (i) token (ii) POS-tag of the token, we can obtain the words that are the important named entities of the document. nltk.ne_chunk_sents returns a tree which has leaves and sub-trees to represent
In [63]:
sentences = sent_tokenize(article)

token_sentences = [ word_tokenize(sentence) for sentence in sentences ]
pprint(token_sentences[:3])

[['Be',
  '‘',
  'a',
  'man',
  'in',
  'the',
  'street',
  'and',
  'a',
  'Jew',
  'in',
  'the',
  'home',
  '’',
  ':',
  'a',
  'common',
  'piece',
  'of',
  'advice',
  'that',
  'liberal',
  'Jews',
  'often',
  'gave',
  'their',
  'co-religionists',
  'in',
  'the',
  '19th',
  'century',
  '.'],
 ['If',
  'Jewishness',
  'was',
  'kept',
  'invisible',
  'and',
  'private',
  ',',
  'they',
  'wagered',
  ',',
  'then',
  'Jews',
  'could',
  'become',
  'citizens',
  'and',
  'professionals',
  ',',
  'and',
  'be',
  'granted',
  'equal',
  'access',
  'to',
  'the',
  'material',
  'resources',
  'made',
  'available',
  'to',
  'any',
  'other',
  'member',
  'of',
  'society',
  '.'],
 ['There',
  'was',
  'plenty',
  'of',
  'Christian',
  'bias',
  'to',
  'combat',
  ',',
  'encapsulated',
  'by',
  'images',
  'of',
  'Jewish',
  'avarice',
  'and',
  'materialism',
  'such',
  'as',
  'Shylock',
  '’',
  's',
  'greedy',
  'hands',
  'and',
  'Rothschild',
  '’',
  's',
  'beard',
  'in',
  'the',
  'form',
  'of',
  'snake-like',
  'tentacles',
  '.']]
In [57]:
pos_sentences = [ nltk.pos_tag(sent) for sent in token_sentences ]
print(pos_sentences[:2])

[[('Be', 'NNP'), ('‘', 'VBZ'), ('a', 'DT'), ('man', 'NN'), ('in', 'IN'), ('the', 'DT'), ('street', 'NN'), ('and', 'CC'), ('a', 'DT'), ('Jew', 'NNP'), ('in', 'IN'), ('the', 'DT'), ('home', 'NN'), ('’', 'NN'), (':', ':'), ('a', 'DT'), ('common', 'JJ'), ('piece', 'NN'), ('of', 'IN'), ('advice', 'NN'), ('that', 'IN'), ('liberal', 'JJ'), ('Jews', 'NNPS'), ('often', 'RB'), ('gave', 'VBD'), ('their', 'PRP$'), ('co-religionists', 'NNS'), ('in', 'IN'), ('the', 'DT'), ('19th', 'JJ'), ('century', 'NN'), ('.', '.')], [('If', 'IN'), ('Jewishness', 'NNP'), ('was', 'VBD'), ('kept', 'VBN'), ('invisible', 'JJ'), ('and', 'CC'), ('private', 'JJ'), (',', ','), ('they', 'PRP'), ('wagered', 'VBD'), (',', ','), ('then', 'RB'), ('Jews', 'NNP'), ('could', 'MD'), ('become', 'VB'), ('citizens', 'NNS'), ('and', 'CC'), ('professionals', 'NNS'), (',', ','), ('and', 'CC'), ('be', 'VB'), ('granted', 'VBN'), ('equal', 'JJ'), ('access', 'NN'), ('to', 'TO'), ('the', 'DT'), ('material', 'NN'), ('resources', 'NNS'), ('made', 'VBD'), ('available', 'JJ'), ('to', 'TO'), ('any', 'DT'), ('other', 'JJ'), ('member', 'NN'), ('of', 'IN'), ('society', 'NN'), ('.', '.')]]
In [75]:
chunked_sentences = nltk.ne_chunk_sents(pos_sentences,binary=True)
print(chunked_sentences)

<generator object ParserI.parse_sents.<locals>.<genexpr> at 0x0000026219631F10>

ne_chunk_sents will convert the list of list of tuples (i.e., list of sentences with tuples of tokens & POS-tags) that is generated by nltk.pos_tag into a generator function. Using a for loop we can access each sentence in the chunked_sentences, and each token and associated POS-tag of each sentence as shown below. Most importantly, it processes the article into named-entity. Having the binary=True is such that named entities are tagged as NE. If binary=False the classifier adds the category labels such as PERSON, ORGANIZATION, GPE, etc. as shown in the Table at the top.

In [110]:
chunked_sentences = nltk.ne_chunk_sents(pos_sentences,binary=True)
print(chunked_sentences)

for sent in chunked_sentences:
    for chunk in sent:
        print(chunk)     

<generator object ParserI.parse_sents.<locals>.<genexpr> at 0x00000262196A6A98>
('Be', 'NNP')
('‘', 'VBZ')
('a', 'DT')
('man', 'NN')
('in', 'IN')
('the', 'DT')
('street', 'NN')
('and', 'CC')
('a', 'DT')
('Jew', 'NNP')
('in', 'IN')
('the', 'DT')
('home', 'NN')
('’', 'NN')
(':', ':')
('a', 'DT')
('common', 'JJ')
('piece', 'NN')
('of', 'IN')
('advice', 'NN')
('that', 'IN')
('liberal', 'JJ')
('Jews', 'NNPS')
('often', 'RB')
('gave', 'VBD')
('their', 'PRP$')
('co-religionists', 'NNS')
('in', 'IN')
('the', 'DT')
('19th', 'JJ')
('century', 'NN')
('.', '.')
('If', 'IN')
(NE Jewishness/NNP)
('was', 'VBD')
('kept', 'VBN')
('invisible', 'JJ')
('and', 'CC')
('private', 'JJ')
(',', ',')
('they', 'PRP')
('wagered', 'VBD')
(',', ',')
('then', 'RB')
('Jews', 'NNP')
('could', 'MD')
('become', 'VB')
('citizens', 'NNS')
('and', 'CC')
('professionals', 'NNS')
(',', ',')
('and', 'CC')
('be', 'VB')
('granted', 'VBN')
('equal', 'JJ')
('access', 'NN')
('to', 'TO')
('the', 'DT')
('material', 'NN')
('resources', 'NNS')
('made', 'VBD')
('available', 'JJ')
('to', 'TO')
('any', 'DT')
('other', 'JJ')
('member', 'NN')
('of', 'IN')
('society', 'NN')
('.', '.')
('There', 'EX')
('was', 'VBD')
('plenty', 'NN')
('of', 'IN')
(NE Christian/JJ)
('bias', 'NN')
('to', 'TO')
('combat', 'VB')
(',', ',')
('encapsulated', 'VBN')
('by', 'IN')
('images', 'NNS')
('of', 'IN')
(NE Jewish/JJ)
('avarice', 'NN')
('and', 'CC')
('materialism', 'NN')
('such', 'JJ')
('as', 'IN')
(NE Shylock/NNP)
('’', 'NNP')
('s', 'VBD')
('greedy', 'JJ')
('hands', 'NNS')
('and', 'CC')
(NE Rothschild/NNP)
('’', 'NNP')
('s', 'VBZ')
('beard', 'NN')
('in', 'IN')
('the', 'DT')
('form', 'NN')
('of', 'IN')
('snake-like', 'JJ')
('tentacles', 'NNS')
('.', '.')
('If', 'IN')
('only', 'RB')
('Jews', 'NNP')
('could', 'MD')
('fit', 'VB')
('into', 'IN')
('the', 'DT')
('spiritual', 'JJ')
('boxes', 'NNS')
('established', 'VBN')
('by', 'IN')
('the', 'DT')
(NE European/NNP Protestant/NNP)
('elite', 'LS')
(',', ',')
('they', 'PRP')
('would', 'MD')
('be', 'VB')
('accepted', 'VBN')
(',', ',')
('or', 'CC')
('at', 'IN')
('least', 'JJS')
('tolerated', 'VBN')
('in', 'IN')
('the', 'DT')
('public', 'NN')
('sphere', 'RB')
('as', 'IN')
(NE Frenchmen/NNP)
(',', ',')
(NE Germans/NNPS)
('or', 'CC')
('Englishmen', 'NNP')
('.', '.')
('Though', 'IN')
('compelling', 'NN')
('in', 'IN')
('theory', 'NN')
(',', ',')
('the', 'DT')
('deal', 'NN')
('became', 'VBD')
('more', 'JJR')
('fraught', 'JJ')
('as', 'IN')
('rampant', 'JJ')
('anti-Semitic', 'JJ')
('violence', 'NN')
('in', 'IN')
('eastern', 'JJ')
(NE Europe/NNP)
('continued', 'VBD')
('to', 'TO')
('remind', 'VB')
('Jews', 'NNP')
('that', 'IN')
(',', ',')
('no', 'DT')
('matter', 'NN')
('how', 'WRB')
('much', 'RB')
('they', 'PRP')
('tried', 'VBD')
('to', 'TO')
('look', 'VB')
('like', 'IN')
('‘', 'NNP')
('everyone', 'NN')
('else', 'RB')
('’', 'NNP')
(',', ',')
('their', 'PRP$')
('bodies', 'NNS')
('were', 'VBD')
('marked', 'VBN')
('as', 'IN')
(NE Jewish/JJ)
('.', '.')
('In', 'IN')
('the', 'DT')
('1870s', 'CD')
(NE European/JJ Judaism/NNP)
('underwent', 'VBD')
('an', 'DT')
('intellectual', 'JJ')
('revolution', 'NN')
('.', '.')
('Around', 'IN')
('then', 'RB')
(',', ',')
('a', 'DT')
('group', 'NN')
('of', 'IN')
('young', 'JJ')
(NE Russian/JJ)
('Jewish', 'JJ')
('radicals', 'NNS')
('began', 'VBD')
('to', 'TO')
('identify', 'VB')
('Judaism', 'NNP')
('with', 'IN')
('materialism', 'NN')
(',', ',')
('and', 'CC')
('to', 'TO')
('theorise', 'VB')
('about', 'IN')
('what', 'WP')
('they', 'PRP')
('called', 'VBD')
('–', 'NNP')
('whether', 'IN')
('in', 'IN')
(NE Russian/NNP)
(',', ',')
(NE German/NNP)
(',', ',')
(NE Yiddish/NNP)
('or', 'CC')
(NE Hebrew/NNP)
('–', 'VBP')
('the', 'DT')
('‘', 'NNP')
('material', 'NN')
('’', 'NNP')
('(', '(')
('material', 'JJ')
('’', 'NNP')
('nii', 'NN')
(',', ',')
('materiell', 'NN')
(',', ',')
('gashmi', 'NN')
(',', ',')
('?', '.')
('omri', 'NN')
(')', ')')
('aspects', 'NNS')
('of', 'IN')
('the', 'DT')
(NE Universe/NNP)
('.', '.')
('For', 'IN')
('many', 'JJ')
('Jews', 'NNPS')
('living', 'VBG')
('in', 'IN')
('this', 'DT')
('period', 'NN')
(',', ',')
('‘', 'JJ')
('materialism', 'NN')
('’', 'NNP')
('was', 'VBD')
('a', 'DT')
('worldview', 'NN')
('that', 'WDT')
('brought', 'VBD')
('into', 'IN')
('focus', 'NN')
('latent', 'JJ')
('Jewish', 'JJ')
('ideas', 'NNS')
('and', 'CC')
('beliefs', 'NNS')
('about', 'IN')
('the', 'DT')
('physical', 'JJ')
('world', 'NN')
('.', '.')
('The', 'DT')
('materialists', 'NNS')
('claimed', 'VBD')
('that', 'IN')
('a', 'DT')
('theory', 'NN')
('of', 'IN')
(NE Judaism/NNP)
(',', ',')
('defined', 'VBN')
('by', 'IN')
('the', 'DT')
('way', 'NN')
('people', 'NNS')
('related', 'VBN')
('to', 'TO')
('land', 'NN')
(',', ',')
('labour', 'NN')
('and', 'CC')
('bodies', 'NNS')
(',', ',')
('had', 'VBD')
('been', 'VBN')
('lying', 'VBG')
('dormant', 'NN')
('within', 'IN')
(NE Jewish/JJ)
('literature', 'NN')
('–', 'NN')
('in', 'IN')
(NE Hasidic/NNP)
('texts', 'NN')
(',', ',')
('the', 'DT')
(NE Bible/NNP)
(',', ',')
(NE Spinoza/NNP)
('’', 'NNP')
('s', 'VBD')
('philosophy', 'NN')
('–', 'NN')
('and', 'CC')
('could', 'MD')
('now', 'RB')
('be', 'VB')
('clearly', 'RB')
('recognised', 'VBN')
('and', 'CC')
('fully', 'RB')
('articulated', 'VBN')
('.', '.')
('Jewish', 'JJ')
('particularity', 'NN')
('was', 'VBD')
('based', 'VBN')
('on', 'IN')
('specific', 'JJ')
('historical', 'JJ')
('economic', 'JJ')
('differences', 'NNS')
('between', 'IN')
('Jews', 'NNP')
('and', 'CC')
('others', 'NNS')
('.', '.')
('What', 'WP')
('made', 'VBD')
('Jews', 'NNP')
('different', 'JJ')
('was', 'VBD')
('a', 'DT')
('certain', 'JJ')
('socioeconomic', 'JJ')
('dynamic', 'NN')
('that', 'WDT')
('distinguished', 'VBD')
('them', 'PRP')
('from', 'IN')
('their', 'PRP$')
('neighbours', 'NNS')
('.', '.')
('The', 'DT')
('Jewish', 'JJ')
('revolutionaries', 'NNS')
('in', 'IN')
('1870s', 'CD')
('Russia', 'NNPS')
('who', 'WP')
('embraced', 'VBD')
('the', 'DT')
('idea', 'NN')
('of', 'IN')
('materialism', 'NN')
('shared', 'VBN')
('a', 'DT')
('number', 'NN')
('of', 'IN')
('critical', 'JJ')
('assumptions', 'NNS')
('.', '.')
('They', 'PRP')
('all', 'DT')
('rejected', 'VBD')
('the', 'DT')
('notion', 'NN')
('that', 'IN')
(NE Judaism/NNP)
('was', 'VBD')
('based', 'VBN')
('on', 'IN')
('abstract', 'JJ')
('metaphysical', 'JJ')
('theories', 'NNS')
('(', '(')
('Scholasticism', 'NNP')
(')', ')')
(',', ',')
('rituals', 'NNS')
('(', '(')
('Hasidism', 'NNP')
(')', ')')
(',', ',')
('study', 'NN')
('(', '(')
(NE Mitnagdim/NNP)
(')', ')')
(',', ',')
('and', 'CC')
('ethics', 'NNS')
('and', 'CC')
('reason', 'NN')
('(', '(')
('Enlighteners', 'NNPS')
(')', ')')
('.', '.')
(NE Judaism/NN)
('was', 'VBD')
('not', 'RB')
('a', 'DT')
('religion', 'NN')
(',', ',')
('like', 'IN')
('Protestantism', 'NNP')
('.', '.')
('Instead', 'RB')
('it', 'PRP')
('was', 'VBD')
('something', 'NN')
('attached', 'VBN')
('to', 'TO')
('their', 'PRP$')
('bodies', 'NNS')
('and', 'CC')
('expressed', 'VBD')
('through', 'IN')
('one', 'CD')
('’', 'NNP')
('s', 'NN')
('relationship', 'NN')
('to', 'TO')
('land', 'NN')
(',', ',')
('labour', 'NN')
('and', 'CC')
('resources', 'NNS')
('.', '.')
('The', 'DT')
('materialists', 'NNS')
('had', 'VBD')
('also', 'RB')
('given', 'VBN')
('up', 'RP')
('hope', 'NN')
('that', 'IN')
('the', 'DT')
('state', 'NN')
('could', 'MD')
('protect', 'VB')
('them', 'PRP')
('and', 'CC')
('ensure', 'VB')
('their', 'PRP$')
('economic', 'JJ')
('wellbeing', 'NN')
('.', '.')
('And', 'CC')
('finally', 'RB')
(',', ',')
('they', 'PRP')
('no', 'RB')
('longer', 'RBR')
('believed', 'VBD')
('that', 'IN')
('history', 'NN')
('was', 'VBD')
('headed', 'VBN')
('in', 'IN')
('a', 'DT')
('positive', 'JJ')
('direction', 'NN')
('.', '.')
('Over', 'IN')
('no', 'DT')
('amount', 'NN')
('of', 'IN')
('time', 'NN')
('would', 'MD')
('Jews', 'VB')
('living', 'NN')
('in', 'IN')
(NE Russia/NNP)
('ever', 'RB')
('be', 'VB')
('granted', 'VBN')
('greater', 'JJR')
('rights', 'NNS')
('and', 'CC')
('opportunities', 'NNS')
('.', '.')
('Therefore', 'RB')
(',', ',')
('only', 'RB')
('a', 'DT')
('radical', 'JJ')
('reclaiming', 'NN')
('of', 'IN')
('the', 'DT')
('physical', 'JJ')
('world', 'NN')
('on', 'IN')
('the', 'DT')
('part', 'NN')
('of', 'IN')
('Jews', 'NNP')
('could', 'MD')
('ensure', 'VB')
('that', 'IN')
('they', 'PRP')
('would', 'MD')
('be', 'VB')
('protected', 'VBN')
('and', 'CC')
('given', 'VBN')
('a', 'DT')
('fair', 'JJ')
('and', 'CC')
('equal', 'JJ')
('share', 'NN')
('of', 'IN')
('resources', 'NNS')
('.', '.')
('Soon', 'RB')
(',', ',')
('the', 'DT')
('Jewish', 'JJ')
('materialism', 'NN')
('of', 'IN')
('the', 'DT')
('Russians', 'NNPS')
('could', 'MD')
('be', 'VB')
('found', 'VBN')
('among', 'IN')
('western', 'JJ')
(NE European/JJ)
('Jews', 'NNPS')
('residing', 'VBG')
('in', 'IN')
(NE England/NNP)
('and', 'CC')
(NE Germany/NNP)
('.', '.')
('Only', 'RB')
('half-jokingly', 'RB')
(',', ',')
('the', 'DT')
(NE German/JJ)
('anarchist', 'NN')
(NE Gustav/NNP Landauer/NNP)
('claimed', 'VBD')
('in', 'IN')
('1921', 'CD')
('that', 'IN')
('what', 'WP')
('distinguished', 'VBD')
('‘', 'VBP')
('the', 'DT')
('modern', 'JJ')
('“', 'NNP')
('conscious', 'JJ')
('”', 'NNP')
('Jew', 'NNP')
('from', 'IN')
('a', 'DT')
(NE German/JJ)
('was', 'VBD')
('that', 'IN')
('when', 'WRB')
('the', 'DT')
('latter', 'NN')
('writes', 'VBZ')
('about', 'IN')
('…', 'PDT')
('the', 'DT')
('conservation', 'NN')
('of', 'IN')
('energy', 'NN')
(',', ',')
('…', 'CC')
('he', 'PRP')
('writes', 'VBZ')
('about', 'IN')
('the', 'DT')
('conservation', 'NN')
('of', 'IN')
('energy', 'NN')
(',', ',')
('but', 'CC')
('when', 'WRB')
('the', 'DT')
('conscious', 'JJ')
('Jew', 'NNP')
('writes', 'VBZ')
('about', 'IN')
('the', 'DT')
('conservation', 'NN')
('of', 'IN')
('energy', 'NN')
(',', ',')
('he', 'PRP')
('writes', 'VBZ')
('about', 'IN')
('the', 'DT')
('conservation', 'NN')
('of', 'IN')
('energy', 'NN')
('and', 'CC')
(NE Judaism/NNP)
('’', 'NNP')
('(', '(')
('emphasis', 'NN')
('mine', 'NN')
(')', ')')
('.', '.')
('Eventually', 'RB')
(',', ',')
('there', 'EX')
('would', 'MD')
('be', 'VB')
('those', 'DT')
(',', ',')
('such', 'JJ')
('as', 'IN')
('the', 'DT')
(NE Englishman/NNP Israel/NNP Zangwill/NNP)
(',', ',')
('who', 'WP')
('considered', 'VBD')
('themselves', 'PRP')
('adherents', 'NNS')
('to', 'TO')
('‘', 'VB')
('a', 'DT')
('religion', 'NN')
('of', 'IN')
('pots', 'NNS')
('and', 'CC')
('pans', 'NNS')
('’', 'VBP')
(',', ',')
('and', 'CC')
('others', 'NNS')
('who', 'WP')
('identified', 'VBD')
('Judaism', 'NNP')
('as', 'IN')
('a', 'DT')
('faith', 'NN')
('based', 'VBN')
('on', 'IN')
('‘', 'NNP')
('bagels', 'NNS')
('and', 'CC')
('lox', 'JJ')
('’', 'NN')
('.', '.')
('Over', 'IN')
('the', 'DT')
('course', 'NN')
('of', 'IN')
('the', 'DT')
('20th', 'JJ')
('century', 'NN')
(',', ',')
(NE Jews/NNP)
('would', 'MD')
('increasingly', 'RB')
('come', 'VB')
('to', 'TO')
('believe', 'VB')
('that', 'IN')
('‘', 'NN')
('there', 'EX')
('is', 'VBZ')
('nothing', 'NN')
('purely', 'RB')
('spiritual', 'JJ')
('that', 'IN')
('stands', 'VBZ')
('on', 'IN')
('its', 'PRP$')
('own', 'JJ')
('…', 'NN')
('Everything', 'NNP')
('spiritual', 'JJ')
('requires', 'VBZ')
('a', 'DT')
('necessary', 'JJ')
('material', 'NN')
('basis.', 'NN')
('’', 'NNP')
('Updates', 'VBZ')
('on', 'IN')
('everything', 'NN')
('new', 'JJ')
('at', 'IN')
(NE Aeon/NNP)
('.', '.')
(NE Top/NN)
('of', 'IN')
(NE Form/NNP Bottom/NNP)
('of', 'IN')
(NE Form/NNP)
('JJjjjJJ', 'NNP')
('Jewish', 'JJ')
('materialists', 'NNS')
('were', 'VBD')
('despised', 'VBN')
('not', 'RB')
('only', 'RB')
('by', 'IN')
('staunch', 'JJ')
('liberals', 'NNS')
('but', 'CC')
('also', 'RB')
('by', 'IN')
('‘', 'JJ')
('defenders', 'NNS')
('of', 'IN')
('the', 'DT')
('faith', 'NN')
('’', 'NN')
('.', '.')
('Moses', 'NNS')
(NE Leib/NNP Lilienblum/NNP)
(',', ',')
('who', 'WP')
('would', 'MD')
('go', 'VB')
('on', 'IN')
('to', 'TO')
('found', 'VB')
('the', 'DT')
(NE Zionist/NNP)
('movement', 'NN')
('in', 'IN')
(NE Russia/NNP)
(',', ',')
('wrote', 'VBD')
('a', 'DT')
('novel', 'NN')
('in', 'IN')
('which', 'WDT')
('he', 'PRP')
('described', 'VBD')
('his', 'PRP$')
('youthful', 'JJ')
('yeshiva', 'JJ')
('education', 'NN')
('as', 'IN')
('one', 'CD')
('long', 'JJ')
('masturbatory', 'NN')
('experience', 'NN')
('–', 'NN')
('for', 'IN')
('this', 'DT')
(',', ',')
('he', 'PRP')
('was', 'VBD')
('denounced', 'VBN')
('by', 'IN')
('rabbis', 'NN')
('and', 'CC')
('communal', 'JJ')
('leaders', 'NNS')
('who', 'WP')
('forced', 'VBD')
('him', 'PRP')
('to', 'TO')
('flee', 'VB')
('his', 'PRP$')
('hometown', 'NN')
('in', 'IN')
('fear', 'NN')
('for', 'IN')
('his', 'PRP$')
('life', 'NN')
('.', '.')
('The', 'DT')
('future', 'NN')
(NE Russian/NNP)
('revolutionary', 'JJ')
(NE Hasia/NNP Schur/NNP)
('was', 'VBD')
('pelted', 'VBN')
('with', 'IN')
('stones', 'NNS')
('and', 'CC')
('jeered', 'VBN')
('at', 'IN')
('by', 'IN')
('the', 'DT')
('townspeople', 'NN')
('of', 'IN')
(NE Mohilev/NNP)
('for', 'IN')
('going', 'VBG')
('on', 'IN')
('a', 'DT')
('Sabbath', 'NNP')
('walk', 'NN')
('hand-in-hand', 'NN')
('with', 'IN')
('her', 'PRP$')
('boyfriend', 'NN')
(',', ',')
('the', 'DT')
('socialist', 'JJ')
(NE Eliezer/NNP)
('Tsukerman', 'NNP')
(':', ':')
('the', 'DT')
('rabbis', 'NN')
('were', 'VBD')
('up', 'RB')
('in', 'IN')
('arms', 'NNS')
('that', 'WDT')
('two', 'CD')
('young', 'JJ')
('people', 'NNS')
('had', 'VBD')
('dared', 'VBN')
('to', 'TO')
('touch', 'VB')
('one', 'CD')
('another', 'DT')
('in', 'IN')
('public', 'NN')
('.', '.')
('Jewish', 'JJ')
('materialists', 'NNS')
('were', 'VBD')
('cast', 'VBN')
('as', 'IN')
('upstarts', 'NNS')
(',', ',')
('deviants', 'NNS')
(',', ',')
('social', 'JJ')
('provocateurs', 'NN')
('and', 'CC')
(',', ',')
('of', 'IN')
('course', 'NN')
(',', ',')
('with', 'IN')
('providing', 'VBG')
('Jew-haters', 'NNS')
('with', 'IN')
('excuses', 'NNS')
('to', 'TO')
('promote', 'VB')
('anti-Semitism', 'NN')
('.', '.')
('But', 'CC')
('the', 'DT')
('Jewish', 'JJ')
('materialists', 'NNS')
('’', 'VBP')
('deviancies', 'NNS')
('reflected', 'VBD')
('a', 'DT')
('radically', 'RB')
('new', 'JJ')
('kind', 'NN')
('of', 'IN')
(NE Jewish/JJ)
('identity', 'NN')
(',', ',')
('one', 'CD')
('focused', 'VBN')
('on', 'IN')
('their', 'PRP$')
('bodies', 'NNS')
('and', 'CC')
('the', 'DT')
('physical', 'JJ')
('world', 'NN')
('.', '.')
('The', 'DT')
('Jewish', 'JJ')
('body', 'NN')
('they', 'PRP')
('imagined', 'VBD')
('would', 'MD')
('offer', 'VB')
('a', 'DT')
('contrast', 'NN')
('to', 'TO')
('both', 'DT')
('the', 'DT')
('hunchbacked', 'VBN')
(',', ',')
('traditional', 'JJ')
('Jewish', 'JJ')
('Torah', 'NNP')
('scholar', 'NN')
('incapable', 'NN')
('of', 'IN')
('supporting', 'VBG')
('his', 'PRP$')
('family', 'NN')
(',', ',')
('and', 'CC')
('the', 'DT')
('muscular', 'JJ')
('gentile', 'NN')
('male', 'NN')
('whose', 'WP$')
('energies', 'NNS')
('were', 'VBD')
('directed', 'VBN')
('at', 'IN')
('conquering', 'VBG')
('and', 'CC')
('dominating', 'VBG')
('the', 'DT')
('physical', 'JJ')
('world', 'NN')
('.', '.')
('The', 'DT')
('new', 'JJ')
('Jewish', 'JJ')
('body', 'NN')
('would', 'MD')
('be', 'VB')
('shaped', 'VBN')
('in', 'IN')
('the', 'DT')
('image', 'NN')
('of', 'IN')
('a', 'DT')
('healthy', 'JJ')
('traditional', 'JJ')
('Jewish', 'JJ')
('woman', 'NN')
('who', 'WP')
('laboured', 'VBD')
('to', 'TO')
('provide', 'VB')
('for', 'IN')
('her', 'PRP$')
('family', 'NN')
('’', 'NN')
('s', 'JJ')
('material', 'NN')
('wellbeing', 'VBG')
('while', 'IN')
('her', 'PRP$')
('husband', 'NN')
('spent', 'VBD')
('his', 'PRP$')
('day', 'NN')
('in', 'IN')
('the', 'DT')
('house', 'NN')
('of', 'IN')
('study', 'NN')
(':', ':')
('by', 'IN')
('tending', 'VBG')
('to', 'TO')
('the', 'DT')
('material', 'JJ')
('aspects', 'NNS')
('of', 'IN')
('existence', 'NN')
(',', ',')
(NE Jews/NNP)
('’', 'NNP')
('needs', 'NNS')
('and', 'CC')
('desires', 'NNS')
('would', 'MD')
('now', 'RB')
('be', 'VB')
('seen', 'VBN')
('as', 'IN')
('the', 'DT')
('primary', 'JJ')
('feature', 'NN')
('of', 'IN')
('Judaism', 'NNP')
('.', '.')
('The', 'DT')
('material', 'NN')
('Jewish', 'JJ')
('identity', 'NN')
('set', 'VBN')
('the', 'DT')
('stage', 'NN')
('for', 'IN')
(NE Jews/NNP)
('’', 'NNP')
('involvement', 'NN')
('in', 'IN')
('20th-century', 'JJ')
('politics', 'NNS')
(':', ':')
('Zionism', 'NN')
(',', ',')
(NE Bundism/NNP)
('(', '(')
('the', 'DT')
('Jewish', 'NNP')
('labour', 'FW')
('movement', 'NN')
(')', ')')
(',', ',')
('the', 'DT')
(NE Minority/NNP Rights/NNP)
('movement', 'NN')
(',', ',')
('and', 'CC')
(NE Jewish/JJ)
('forms', 'NNS')
('of', 'IN')
('communism', 'NN')
('all', 'DT')
('assumed', 'VBD')
('that', 'IN')
('the', 'DT')
('organising', 'JJ')
('structure', 'NN')
('of', 'IN')
(NE Jewish/JJ)
('identity', 'NN')
('was', 'VBD')
('a', 'DT')
('Jewish', 'JJ')
('body', 'NN')
(',', ',')
('and', 'CC')
('not', 'RB')
('a', 'DT')
('Judaism', 'NN')
('of', 'IN')
('the', 'DT')
('heavens', 'NNS')
('or', 'CC')
('the', 'DT')
('heart', 'NN')
('.', '.')
('Jewish', 'JJ')
('materialism', 'NN')
('made', 'VBD')
('Jews', 'NNP')
('political', 'JJ')
('without', 'IN')
('them', 'PRP')
('possessing', 'VBG')
('their', 'PRP$')
('own', 'JJ')
('state', 'NN')
('or', 'CC')
('even', 'RB')
('citizenship', 'NN')
('in', 'IN')
('a', 'DT')
('host', 'NN')
('country', 'NN')
('.', '.')
('Though', 'IN')
('the', 'DT')
('idea', 'NN')
('of', 'IN')
('the', 'DT')
('Jewish', 'JJ')
('body', 'NN')
('as', 'IN')
('the', 'DT')
('locus', 'NN')
('of', 'IN')
('collective', 'JJ')
('identity', 'NN')
('would', 'MD')
('always', 'RB')
('be', 'VB')
('suspect', 'JJ')
('in', 'IN')
('western', 'JJ')
(NE Europe/NNP)
(',', ',')
('it', 'PRP')
('would', 'MD')
(',', ',')
('however', 'RB')
(',', ',')
('become', 'VBP')
('the', 'DT')
('basis', 'NN')
('of', 'IN')
('a', 'DT')
('new', 'JJ')
('kind', 'NN')
('of', 'IN')
(NE Jewish/JJ)
('identity', 'NN')
('most', 'RBS')
('commonly', 'RB')
('witnessed', 'VBN')
('in', 'IN')
(NE Israel/NNP)
('and', 'CC')
('the', 'DT')
(NE United/NNP States/NNPS)
('.', '.')
('Jewish', 'JJ')
('immigrants', 'NNS')
('to', 'TO')
('Palestine', 'VB')
('at', 'IN')
('the', 'DT')
('turn', 'NN')
('of', 'IN')
('the', 'DT')
('century', 'NN')
('saw', 'VBD')
('in', 'IN')
('Zion', 'NNP')
('the', 'DT')
('actualisation', 'NN')
('of', 'IN')
('materialism', 'NN')
('as', 'IN')
('first', 'JJ')
('imagined', 'VBN')
('in', 'IN')
('the', 'DT')
('1870s', 'CD')
('.', '.')
('The', 'DT')
(NE Marxist/JJ Ber/NNP Borchov/NNP)
('’', 'NNP')
('s', 'NN')
('students', 'NNS')
(',', ',')
('such', 'JJ')
('as', 'IN')
('future', 'JJ')
('leaders', 'NNS')
('of', 'IN')
(NE Israel/NNP Yitzhak/NNP)
('Ben-Zvi', 'NNP')
('and', 'CC')
(NE David/NNP Ben-Gurion/NNP)
(',', ',')
('identified', 'JJ')
('Palestine', 'NNP')
('as', 'IN')
('a', 'DT')
('response', 'NN')
('to', 'TO')
('the', 'DT')
('crisis', 'NN')
('of', 'IN')
('the', 'DT')
('fork', 'NN')
('and', 'CC')
('the', 'DT')
('knife', 'NN')
('(', '(')
('a', 'DT')
('pithy', 'NN')
('phrase', 'NN')
('meant', 'VBD')
('to', 'TO')
('capture', 'VB')
('the', 'DT')
('economic', 'JJ')
('challenges', 'NNS')
('of', 'IN')
(NE Russian/JJ)
('Jews', 'NNPS')
('in', 'IN')
('the', 'DT')
('1870s', 'CD')
(')', ')')
('originally', 'RB')
('theorised', 'VBN')
('by', 'IN')
('the', 'DT')
('Jewish', 'JJ')
('materialist', 'NN')
(NE Aaron/NNP Shemuel/NNP Lieberman/NNP)
('in', 'IN')
('the', 'DT')
('1870s', 'CD')
('.', '.')
('They', 'PRP')
('envisioned', 'VBD')
('a', 'DT')
('new', 'JJ')
('kind', 'NN')
('of', 'IN')
(NE Jew/NNP)
('–', 'NNP')
('the', 'DT')
('?', '.')
('aluts', 'NNS')
('(', '(')
('pioneer', 'NN')
(')', ')')
('–', 'NN')
('who', 'WP')
('was', 'VBD')
('attached', 'VBN')
('to', 'TO')
('the', 'DT')
('physical', 'JJ')
('world', 'NN')
('.', '.')
('As', 'IN')
('described', 'VBN')
('by', 'IN')
('the', 'DT')
('20th-century', 'JJ')
('Zionist', 'NNP')
('poet', 'NN')
(NE Avraham/NNP Shlonsky/NNP)
(',', ',')
('a', 'DT')
('former', 'JJ')
(NE Hasidic/NNP Jew/NNP)
(',', ',')
('the', 'DT')
('?', '.')
('aluts', 'NNS')
('would', 'MD')
('be', 'VB')
('the', 'DT')
('embodiment', 'NN')
('of', 'IN')
('the', 'DT')
('idea', 'NN')
('that', 'WDT')
('‘', 'VBZ')
('a', 'DT')
('human', 'JJ')
('being', 'VBG')
('is', 'VBZ')
('meat', 'NN')
(',', ',')
('and', 'CC')
('he', 'PRP')
('toils', 'VBZ')
('here', 'RB')
('in', 'IN')
('the', 'DT')
('sacred/and', 'NN')
('the', 'DT')
('land/bread', 'JJ')
('’', 'NN')
('.', '.')
('The', 'DT')
('people', 'NNS')
('of', 'IN')
('the', 'DT')
('book', 'NN')
('had', 'VBD')
('now', 'RB')
('become', 'VBN')
('a', 'DT')
('people', 'NNS')
('of', 'IN')
('labour', 'NN')
(',', ',')
('land', 'NN')
('and', 'CC')
('the', 'DT')
('body', 'NN')
('.', '.')
('In', 'IN')
('the', 'DT')
(NE US/NNP)
(',', ',')
('eastern', 'JJ')
(NE European/JJ Jews/NNP)
('established', 'VBD')
('large-scale', 'JJ')
('defence', 'NN')
('organisations', 'NNS')
('directed', 'VBN')
('at', 'IN')
('protecting', 'VBG')
('Jewish', 'JJ')
('bodies', 'NNS')
('and', 'CC')
('providing', 'VBG')
('a', 'DT')
('platform', 'NN')
('for', 'IN')
('Jews', 'NNP')
('to', 'TO')
('speak', 'VB')
('as', 'IN')
('a', 'DT')
('distinct', 'JJ')
('ethnic', 'JJ')
('minority', 'NN')
('in', 'IN')
('the', 'DT')
(NE American/JJ)
('public', 'NN')
('sphere', 'RB')
('.', '.')
('From', 'IN')
('the', 'DT')
('poet', 'NN')
(NE Emma/NNP Lazarus/NNP)
('to', 'TO')
('the', 'DT')
(NE American/JJ)
('rabbi', 'NN')
('Mordecai', 'NNP')
('Kaplan', 'NNP')
('to', 'TO')
('the', 'DT')
('philosopher', 'NN')
(NE Horace/NNP Kallen/NNP)
(',', ',')
(NE American/NNP Jews/NNP)
('in', 'IN')
('the', 'DT')
('early', 'JJ')
('20th', 'JJ')
('century', 'NN')
('developed', 'VBD')
('political', 'JJ')
('programmes', 'NNS')
('and', 'CC')
('established', 'VBN')
('organisations', 'NNS')
('rooted', 'VBN')
('around', 'IN')
('the', 'DT')
('physical', 'JJ')
('aspects', 'NNS')
('of', 'IN')
(NE Jewish/JJ)
('life', 'NN')
('.', '.')
('Jewish', 'JJ')
('materialism', 'NN')
('remains', 'VBZ')
('the', 'DT')
('defining', 'VBG')
('element', 'NN')
('of', 'IN')
('most', 'JJS')
(NE American/JJ Jews/NNP)
('’', 'NNP')
('identity', 'NN')
('.', '.')
('Following', 'VBG')
('the', 'DT')
('Second', 'JJ')
('World', 'NNP')
('War', 'NNP')
(',', ',')
('the', 'DT')
('influx', 'NN')
('of', 'IN')
('another', 'DT')
('wave', 'NN')
('of', 'IN')
(NE Jewish/JJ)
('immigrants', 'NNS')
('from', 'IN')
(NE Russian/JJ)
('lands', 'NNS')
('gave', 'VBD')
('rise', 'NN')
('to', 'TO')
('a', 'DT')
('new', 'JJ')
('brand', 'NN')
('of', 'IN')
(NE US/NNP)
('literature', 'NN')
('that', 'WDT')
('placed', 'VBD')
('the', 'DT')
('Jewish', 'JJ')
('body', 'NN')
('front', 'NN')
('and', 'CC')
('centre', 'NN')
('.', '.')
('The', 'DT')
('late', 'JJ')
(NE US/NNP)
('novelist', 'NN')
(NE Phillip/NNP Roth/NNP)
('might', 'MD')
('have', 'VB')
('been', 'VBN')
('familiar', 'JJ')
('only', 'RB')
('in', 'IN')
('passing', 'VBG')
('with', 'IN')
('the', 'DT')
('name', 'NN')
(NE Moses/NNP Lilienblum/NNP)
('.', '.')
('But', 'CC')
('it', 'PRP')
('was', 'VBD')
('Lilienblum', 'NNP')
('who', 'WP')
('put', 'VBD')
('into', 'IN')
('circulation', 'NN')
('the', 'DT')
('Jewish', 'JJ')
('genre', 'NN')
('of', 'IN')
('overbearing', 'VBG')
('parents', 'NNS')
(',', ',')
('unrealisable', 'JJ')
('social', 'JJ')
('expectations', 'NNS')
(',', ',')
('failed', 'VBD')
('sexual', 'JJ')
('encounters', 'NNS')
(',', ',')
('silly', 'RB')
('rabbis', 'NN')
(',', ',')
('bankrupt', 'JJ')
('synagogues', 'NNS')
('and', 'CC')
('God-fearing', 'NNP')
('charlatans', 'NNS')
('encased', 'VBN')
('in', 'IN')
('a', 'DT')
('narrative', 'JJ')
('about', 'IN')
('masturbation', 'NN')
('.', '.')
('Whether', 'IN')
('he', 'PRP')
('knew', 'VBD')
('it', 'PRP')
('or', 'CC')
('not', 'RB')
(',', ',')
('when', 'WRB')
(NE Roth/NNP)
('wrote', 'VBD')
('his', 'PRP$')
('novel', 'JJ')
(NE Portnoy/NNP)
('’', 'NNP')
('s', 'PRP')
('Complaint', 'NNP')
('(', '(')
('1969', 'CD')
(')', ')')
(',', ',')
('he', 'PRP')
('was', 'VBD')
('channelling', 'VBG')
('the', 'DT')
('same', 'JJ')
('tradition', 'NN')
('first', 'RB')
('articulated', 'VBN')
('by', 'IN')
('Lilienblum', 'NNP')
('a', 'DT')
('century', 'NN')
('earlier', 'RBR')
('.', '.')
(NE Roth/NNP)
('took', 'VBD')
('those', 'DT')
('commitments', 'NNS')
('to', 'TO')
('his', 'PRP$')
('grave', 'NN')
('when', 'WRB')
('he', 'PRP')
('died', 'VBD')
('on', 'IN')
('22', 'CD')
('May', 'NNP')
('2018', 'CD')
('.', '.')
('While', 'IN')
('the', 'DT')
('grandmaster', 'NN')
('of', 'IN')
('late-20th-century', 'JJ')
(NE American/JJ)
('letters', 'NNS')
('asked', 'VBD')
('to', 'TO')
('be', 'VB')
('interred', 'VBN')
('next', 'JJ')
('to', 'TO')
('Jews', 'NNPS')
(',', ',')
('he', 'PRP')
('strictly', 'RB')
('prohibited', 'VBD')
('the', 'DT')
('performance', 'NN')
('of', 'IN')
('any', 'DT')
('Jewish', 'JJ')
('rituals', 'NNS')
('at', 'IN')
('his', 'PRP$')
('funeral', 'NN')
('.', '.')
('His', 'PRP$')
('final', 'JJ')
('requests', 'NNS')
(',', ',')
('allegedly', 'RB')
(',', ',')
('were', 'VBD')
('inspired', 'VBN')
('by', 'IN')
('a', 'DT')
('desire', 'NN')
('‘', 'NN')
('to', 'TO')
('have', 'VB')
('someone', 'NN')
('to', 'TO')
('talk', 'VB')
('to', 'TO')
('’', 'VB')
('.', '.')
('His', 'PRP$')
('corpse', 'NN')
('did', 'VBD')
('not', 'RB')
('need', 'VB')
('a', 'DT')
('rabbi', 'NN')
('to', 'TO')
('eulogise', 'VB')
('it', 'PRP')
(',', ',')
('or', 'CC')
('a', 'DT')
('perfunctory', 'JJ')
('kaddish', 'NN')
('(', '(')
('or', 'CC')
('hymn', 'NN')
(')', ')')
('to', 'TO')
('kasher', 'VB')
('it', 'PRP')
(';', ':')
('it', 'PRP')
('was', 'VBD')
('simply', 'RB')
('Jewish', 'JJ')
('–', 'NN')
('nothing', 'NN')
('more', 'RBR')
('and', 'CC')
('nothing', 'NN')
('less', 'JJR')
('.', '.')
('Indeed', 'RB')
(',', ',')
('it', 'PRP')
('was', 'VBD')
('a', 'DT')
('fitting', 'JJ')
('conclusion', 'NN')
('to', 'TO')
('the', 'DT')
('life', 'NN')
('of', 'IN')
('a', 'DT')
('Jewish', 'JJ')
('materialist', 'NN')
('.', '.')

We can see below that any chunks that have a label and is a NE that is a named-entity we print it out. We find that only certain chunks in our document are named entities.

In [117]:
chunked_sentences = nltk.ne_chunk_sents(pos_sentences,binary=True)

for sent in chunked_sentences:
    for chunk in sent:
        if hasattr(chunk,'label') and chunk.label()=='NE':
            print(chunk)  

(NE Jewishness/NNP)
(NE Christian/JJ)
(NE Jewish/JJ)
(NE Shylock/NNP)
(NE Rothschild/NNP)
(NE European/NNP Protestant/NNP)
(NE Frenchmen/NNP)
(NE Germans/NNPS)
(NE Europe/NNP)
(NE Jewish/JJ)
(NE European/JJ Judaism/NNP)
(NE Russian/JJ)
(NE Russian/NNP)
(NE German/NNP)
(NE Yiddish/NNP)
(NE Hebrew/NNP)
(NE Universe/NNP)
(NE Judaism/NNP)
(NE Jewish/JJ)
(NE Hasidic/NNP)
(NE Bible/NNP)
(NE Spinoza/NNP)
(NE Judaism/NNP)
(NE Mitnagdim/NNP)
(NE Judaism/NN)
(NE Russia/NNP)
(NE European/JJ)
(NE England/NNP)
(NE Germany/NNP)
(NE German/JJ)
(NE Gustav/NNP Landauer/NNP)
(NE German/JJ)
(NE Judaism/NNP)
(NE Englishman/NNP Israel/NNP Zangwill/NNP)
(NE Jews/NNP)
(NE Aeon/NNP)
(NE Top/NN)
(NE Form/NNP Bottom/NNP)
(NE Form/NNP)
(NE Leib/NNP Lilienblum/NNP)
(NE Zionist/NNP)
(NE Russia/NNP)
(NE Russian/NNP)
(NE Hasia/NNP Schur/NNP)
(NE Mohilev/NNP)
(NE Eliezer/NNP)
(NE Jewish/JJ)
(NE Jews/NNP)
(NE Jews/NNP)
(NE Bundism/NNP)
(NE Minority/NNP Rights/NNP)
(NE Jewish/JJ)
(NE Jewish/JJ)
(NE Europe/NNP)
(NE Jewish/JJ)
(NE Israel/NNP)
(NE United/NNP States/NNPS)
(NE Marxist/JJ Ber/NNP Borchov/NNP)
(NE Israel/NNP Yitzhak/NNP)
(NE David/NNP Ben-Gurion/NNP)
(NE Russian/JJ)
(NE Aaron/NNP Shemuel/NNP Lieberman/NNP)
(NE Jew/NNP)
(NE Avraham/NNP Shlonsky/NNP)
(NE Hasidic/NNP Jew/NNP)
(NE US/NNP)
(NE European/JJ Jews/NNP)
(NE American/JJ)
(NE Emma/NNP Lazarus/NNP)
(NE American/JJ)
(NE Horace/NNP Kallen/NNP)
(NE American/NNP Jews/NNP)
(NE Jewish/JJ)
(NE American/JJ Jews/NNP)
(NE Jewish/JJ)
(NE Russian/JJ)
(NE US/NNP)
(NE US/NNP)
(NE Phillip/NNP Roth/NNP)
(NE Moses/NNP Lilienblum/NNP)
(NE Roth/NNP)
(NE Portnoy/NNP)
(NE Roth/NNP)
(NE American/JJ)

The binary keyword for nltk.ne_chunk_sents

When we set binary=False, the classifier adds cateory labels to the named entities. Thus we do not just have ne but we havev more informative labels such as GPE, PERSON, ORGANIZATION, etc.

In [119]:
chunked_sentences = nltk.ne_chunk_sents(pos_sentences,binary=False)

for sent in chunked_sentences:
    for chunk in sent:
        if hasattr(chunk,'label'):
            print(chunk)

(GPE Be/NNP)
(PERSON Jewishness/NNP)
(PERSON Jews/NNP)
(GPE Christian/JJ)
(GPE Jewish/JJ)
(PERSON Shylock/NNP)
(PERSON Rothschild/NNP)
(PERSON Jews/NNP)
(ORGANIZATION European/NNP)
(GPE Frenchmen/NNP)
(GPE Germans/NNPS)
(GPE Englishmen/NNP)
(PERSON Europe/NNP)
(PERSON Jews/NNP)
(GPE Jewish/JJ)
(ORGANIZATION European/JJ Judaism/NNP)
(GPE Russian/JJ)
(GPE Russian/NNP)
(GPE German/NNP)
(GPE Yiddish/NNP)
(GPE Hebrew/NNP)
(ORGANIZATION Universe/NNP)
(GPE Judaism/NNP)
(GPE Jewish/JJ)
(GPE Hasidic/NNP)
(ORGANIZATION Bible/NNP)
(PERSON Spinoza/NNP)
(GPE Jewish/JJ)
(PERSON Jews/NNP)
(PERSON Jews/NNP)
(GPE Jewish/JJ)
(PERSON Judaism/NNP)
(ORGANIZATION Mitnagdim/NNP)
(PERSON Judaism/NN)
(GPE Russia/NNP)
(PERSON Jews/NNP)
(GSP Russians/NNPS)
(ORGANIZATION European/JJ Jews/NNPS)
(GPE England/NNP)
(GPE Germany/NNP)
(GPE German/JJ)
(PERSON Gustav/NNP Landauer/NNP)
(GPE German/JJ)
(PERSON Judaism/NNP)
(ORGANIZATION Englishman/NNP Israel/NNP Zangwill/NNP)
(PERSON Jews/NNP)
(ORGANIZATION Aeon/NNP)
(ORGANIZATION Form/NNP Bottom/NNP)
(ORGANIZATION Form/NNP)
(PERSON Leib/NNP Lilienblum/NNP)
(ORGANIZATION Zionist/NNP)
(GPE Russia/NNP)
(GPE Russian/NNP)
(PERSON Hasia/NNP Schur/NNP)
(GPE Mohilev/NNP)
(PERSON Eliezer/NNP Tsukerman/NNP)
(GPE Jewish/JJ)
(GPE Jewish/JJ)
(GPE Jewish/JJ)
(PERSON Jews/NNP)
(GPE Judaism/NNP)
(PERSON Jews/NNP)
(GPE Bundism/NNP)
(ORGANIZATION Minority/NNP Rights/NNP)
(GPE Jewish/JJ)
(GPE Jewish/JJ)
(GPE Jewish/JJ)
(PERSON Jews/NNP)
(GPE Europe/NNP)
(GPE Jewish/JJ)
(GPE Israel/NNP)
(GPE United/NNP States/NNPS)
(GPE Jewish/JJ)
(ORGANIZATION Marxist/JJ Ber/NNP Borchov/NNP)
(GPE Israel/NNP)
(PERSON Yitzhak/NNP Ben-Zvi/NNP)
(PERSON David/NNP)
(GPE Russian/JJ)
(PERSON Aaron/NNP Shemuel/NNP Lieberman/NNP)
(PERSON Jew/NNP)
(PERSON Avraham/NNP Shlonsky/NNP)
(ORGANIZATION Hasidic/NNP Jew/NNP)
(GSP US/NNP)
(ORGANIZATION European/JJ Jews/NNP)
(GPE Jewish/JJ)
(PERSON Jews/NNP)
(GPE American/JJ)
(PERSON Emma/NNP Lazarus/NNP)
(GPE American/JJ)
(PERSON Mordecai/NNP Kaplan/NNP)
(PERSON Horace/NNP Kallen/NNP)
(GPE American/NNP)
(GPE Jewish/JJ)
(GPE Jewish/JJ)
(ORGANIZATION American/JJ Jews/NNP)
(ORGANIZATION Second/JJ)
(GPE Jewish/JJ)
(GPE Russian/JJ)
(GSP US/NNP)
(GSP US/NNP)
(PERSON Phillip/NNP Roth/NNP)
(PERSON Moses/NNP Lilienblum/NNP)
(PERSON Roth/NNP)
(PERSON Portnoy/NNP)
(ORGANIZATION Lilienblum/NNP)
(PERSON Roth/NNP)
(GPE American/JJ)

Printing out percentage of each cateogry

In [164]:
chunked_sentences = nltk.ne_chunk_sents(pos_sentences,binary=False)


ner_categories=collections.defaultdict(int)

for sent in chunked_sentences:
    for chunk in sent:
        if hasattr(chunk,'label'):
            ner_categories[chunk.label()] += 1
            
print(ner_categories)

labels = list(ner_categories.keys())
values = [ner_categories.get(l) for l in labels]

plt.pie(values,labels=labels)

defaultdict(<class 'int'>, {'GPE': 49, 'PERSON': 36, 'ORGANIZATION': 18, 'GSP': 4})
Out[164]:
([<matplotlib.patches.Wedge at 0x2621d7e1b70>,
  <matplotlib.patches.Wedge at 0x2621d7e9080>,
  <matplotlib.patches.Wedge at 0x2621d7e9550>,
  <matplotlib.patches.Wedge at 0x2621d7e9a90>],
 [Text(0.144913,1.09041,'GPE'),
  Text(-0.772087,-0.783506,'PERSON'),
  Text(0.794755,-0.760502,'ORGANIZATION'),
  Text(1.09242,-0.12889,'GSP')])
In [165]:
plt.pie(values,labels=labels,autopct='%1.1f%%',startangle=90)
Out[165]:
([<matplotlib.patches.Wedge at 0x2621d821780>,
  <matplotlib.patches.Wedge at 0x2621d821e48>,
  <matplotlib.patches.Wedge at 0x2621d82a630>,
  <matplotlib.patches.Wedge at 0x2621d82add8>],
 [Text(-1.09041,0.144913,'GPE'),
  Text(0.783506,-0.772087,'PERSON'),
  Text(0.760502,0.794755,'ORGANIZATION'),
  Text(0.12889,1.09242,'GSP')],
 [Text(-0.594771,0.0790434,'45.8%'),
  Text(0.427367,-0.421139,'33.6%'),
  Text(0.414819,0.433503,'16.8%'),
  Text(0.0703037,0.595867,'3.7%')])

Comments

Comments powered by Disqus