Download Natural Language Processing in Python and more Thesis Computer Science in PDF only on Docsity!
Natural Language Processing (NLP) in
Python – Complete Study Notes
Introduction
NLP is a branch of AI that focuses on enabling machines to understand, interpret, and generate human language. It is widely used in applications such as chatbots, translation systems, and sentiment analysis. Python provides powerful libraries like NLTK and spaCy for NLP tasks.
Text Preprocessing
Text preprocessing prepares raw text for analysis. It includes steps such as tokenization, stopword removal, and stemming. Example Code: from nltk.tokenize import word_tokenize text = 'NLP is interesting' print(word_tokenize(text))
Tokenization
Tokenization splits text into words or sentences. It is the first step in NLP pipelines. It helps in analyzing text structure.
Stopwords Removal
Stopwords are common words like 'is', 'the', 'and'. They are removed to focus on meaningful words. Example Code: from nltk.corpus import stopwords stop_words = set(stopwords.words('english'))
Stemming and Lemmatization
Stemming reduces words to root form. Lemmatization converts words to meaningful base forms. Example Code: from nltk.stem import PorterStemmer ps = PorterStemmer() print(ps.stem('running'))
Bag of Words
Bag of Words represents text as frequency of words. It ignores grammar but keeps important terms. It is widely used in text classification.
TF-IDF
TF-IDF measures importance of words in documents. It reduces weight of common words and increases rare words. Used in search engines and document ranking.
Sentiment Analysis
Sentiment analysis determines emotion behind text. It classifies text as positive, negative, or neutral. Example Code: from textblob import TextBlob analysis = TextBlob('I love this product') print(analysis.sentiment)
Named Entity Recognition (NER)
NER identifies entities such as names, locations, and organizations. It is used in information extraction. Example Code:
from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer(ngram_range=(1,2)) X = vectorizer.fit_transform(['I love NLP']) print(vectorizer.get_feature_names_out())
Text Classification
Text classification involves assigning categories to text data. It is widely used in spam detection and sentiment analysis. Example Code: from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB texts = ['I love this', 'I hate this'] labels = [1, 0] vec = TfidfVectorizer() X = vec.fit_transform(texts) model = MultinomialNB() model.fit(X, labels)
Topic Modeling
Topic modeling identifies hidden topics in a collection of documents. Latent Dirichlet Allocation (LDA) is commonly used. It helps in organizing large text datasets. Example Code: from sklearn.decomposition import LatentDirichletAllocation lda = LatentDirichletAllocation(n_components=2) lda.fit(X)
Mini Project: Spam Detection
A simple NLP project is email spam detection. Steps: preprocess text, convert using TF-IDF, train classifier.
This demonstrates real-world NLP usage. Example Code: from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.2) model.fit(X_train, y_train) print(model.score(X_test, y_test))
