AI - Use Natural Language Toolkit to split up text by word

 

A simple way to get words from text by using NLTK, Natural Language Toolkit

Code :

from nltk.tokenize import sent_tokenize, word_tokenize

text = """

Do not let your hearts be troubled. You believe in God[a]; believe also in me.

My Father’s house has many rooms; if that were not so, would I have told you that I am going there to prepare a place for you?

And if I go and prepare a place for you, I will come back and take you to be with me that you also may be where I am.

You know the way to the place where I am going.

Thomas said to him, “Lord, we don’t know where you are going, so how can we know the way?”

Jesus answered, I am the way and the truth and the life.

No one comes to the Father except through me.

If you really know me, you will know[b] my Father as well. From now on, you do know him and have seen him.

"""

arrayString =  sent_tokenize(text)

arrayWords = word_tokenize(text)

# print(arrayString)

print(arrayWords)

Result:

(base) C:\>python C:\Projects\Python\NLTK\Tokenizing.py

['Do', 'not', 'let', 'your', 'hearts', 'be', 'troubled', '.', 'You', 'believe', 'in', 'God', '[', 'a', ']', ';', 'believe', 'also', 'in', 'me', '.', 'My', 'Father', '’', 's', 'house', 'has', 'many', 'rooms', ';', 'if', 'that', 'were', 'not', 'so', ',', 'would', 'I', 'have', 'told', 'you', 'that', 'I', 'am', 'going', 'there', 'to', 'prepare', 'a', 'place', 'for', 'you', '?', 'And', 'if', 'I', 'go', 'and', 'prepare', 'a', 'place', 'for', 'you', ',', 'I', 'will', 'come', 'back', 'and', 'take', 'you', 'to', 'be', 'with', 'me', 'that', 'you', 'also', 'may', 'be', 'where', 'I', 'am', '.', 'You', 'know', 'the', 'way', 'to', 'the', 'place', 'where', 'I', 'am', 'going', '.', 'Thomas', 'said', 'to', 'him', ',', '“', 'Lord', ',', 'we', 'don', '’', 't', 'know', 'where', 'you', 'are', 'going', ',', 'so', 'how', 'can', 'we', 'know', 'the', 'way', '?', '”', 'Jesus', 'answered', ',', 'I', 'am', 'the', 'way', 'and', 'the', 'truth', 'and', 'the', 'life', '.', 'No', 'one', 'comes', 'to', 'the', 'Father', 'except', 'through', 'me', '.', 'If', 'you', 'really', 'know', 'me', ',', 'you', 'will', 'know', '[', 'b', ']', 'my', 'Father', 'as', 'well', '.', 'From', 'now', 'on', ',', 'you', 'do', 'know', 'him', 'and', 'have', 'seen', 'him', '.']

My services: Predict and more detail in each group/cluster, between each cluster/group or do predict in number of stock codes or do predict in any stock data from any stock market, (US, Hong Kong, Singapore, Japan, London, Korea) .

You need to get an AI, Machine Learning or OpenAI system? Call me!

Cut 90% cost by using my development services for AI, Machine Learning, Mobile App and Web App!

Call me: +84854147015

WhatsApp: +601151992689

https://amatasiam.web.app

Email: ThomasTrungVo@Gmail.Com