Download Data Mining Technologies in Business Intelligence: INFS 2036 Week 8 Notes and more Lecture notes Business in PDF only on Docsity!
BI
BUSINESS INTELLIGENCE
INFS 2036
Business Intelligence
We e k 8
T H E N E E D F O R +
A P P L I C A B I L I T Y O F
D ATA M I N I N G T E C H N O L O G I E S
Course Outline
INFS 2036 2
The Why
of BI
The How of BI
Emerging
BI Concepts
Future Analytics
Week 1 Why data is important to business + The BI Process Week 2 Front-End: Data Visualisation Week 3 Back-End: Organisation Information Systems Week 4 From Data to Intelligence Week 5 Lifecycle model and project management Week 6 Analytics - predicting the future & performance management Week 7 Privacy, Ethics, Legal Issues, Trust Week 8 Mining Technologies Week 9 Data Integration at an Enterprise and Cross-Organisation Level + Strategic value of information in the future You’re the CEO (Exam Review) Week 10
Week 8 Reading/Viewing Textbook
- Chapter 4: 4.5 Data Mining Methods (pages 241-257)
- Chapter 5: 5.2 Text analytics and Text Mining Overview to 5.4 Text mining applications (pages 277-294)
- Chapter 5: 5.6 Sentiment Analysis (up to but not including Sentiment Analysis Process) (pages 302-308)
- Chapter 5: 5.10. Social Analytics (pages 330-339)
- Chapter 6: 6.1. Opening Vignette: School District of Philadelphia Uses Prescriptive Analytics to Find Optimal Solution for Awarding Bus Route Contracts (pages 346-347) INFS 2036 3 Additional reading/viewing – see Workshop Notes page (course site) HBR: Stitch Fix’s CEO on Selling Personal Style to the Mass Market: https://hbr.org/2018/05/stitch-fixs- ceo-on-selling-personal-style-to-the-mass-market
- Focus On … Modelling + Predicting
Customer Behaviour via Classification
- Putting it Into Practice
- What + have in common
- What’s coming next… Key Points + Workshop Activities
The need for data mining technologies in
competitive business environments:
- Market Basket Analysis
- Association Rule Mining
- Confidence, Lift + Support
- Cross-Sell + Up-Sell via Recommender Systems
- Two-Class + Multi-Class Classification Topic Presentation + Practice W h a t w e ’ r e d o i n g t h i s w e e k INFS 2036 4 Agenda – Week 8 THEORY PRACTICE This Week’s Case Study
Online retailers wanting
to increase sales through
recommendations
INFS 2036 7
We’ve seen visualisation and
next week we’ll see a few
more tools.
Today we’ll focus on Data
Mining and Predictive
Analytics
https://paristech.com/blog/investing-in-information/
Why Data Mining? INFS 2036 8
Why Data Mining? INFS 2036 9 Being able to use the information you gather is at least as important as gathering it.
- Anyone performing analytics
- In Industry 4.
Data Mining vs BI vs Data Science 10 Data Science: Data Mining Business Intelligence break down information build up information Analysis Synthesis Y = $ + &!×(! +&"×("
D E S C R I P T I V E + P R E D I C T I V E
Data Mining INFS 2036 13
DESCRIPTIVE
ANALYTICS
PREDICTIVE
ANALYTICS
- What happened in the past – uses existing data
- Uses data aggregation and data mining
- Find information by analysing big data
- Uses past data for presentation and visualisation
- Produces quantifiable information e.g. sales, performance analysis, etc.
- Provides accurate data in reports
- Supports a reactive approach in business (Business Intelligence) (Business Analytics) - Uses past data to forecast the future - Uses statistics and forecast techniques - Forecasts future information by analysing big data - Uses past data to build statistical forecasting models - Forecasts quantifiable and qualitative information e.g. sales, sentiment analysis etc. - Forecasts give insight into what might happen - Supports a proactive approach in business
Recommender Systems INFS 2036 14
C R O S S - S E L L + U P - S E L L
- Recommender systems learn patterns of behaviours to predict (forecast) a person’s preference on a set of items they have not yet experienced.
- A main goal of recommender systems is product up-sell and/or cross-sell.
- Cross-sell: suggest additional products for customers to purchase.
- Up-sell: suggest upgraded and/or additional products for customers to purchase.
- Many companies use recommender systems to cross- and up-sell:
- Products to purchase (e.g. Amazon)
- Movies to watch — e.g. Netflix (awarded $1,000,000 to the team that improved their predictions)
- Music to purchase or listen to — iTunes shows Related music (e.g. Ed Sheeran’s back catalogue)
- Spotify suggests a tailored playlist of 30 songs a listener has never heard before
- People to connect with. If recommendations are successful, the average order size per customer should increase, thus increasing revenue. DESCRIPTIVE ANALYTICS PREDICTIVE ANALYTICS
INFS 2036 15
C R O S S - S E L L + U P - S E L L
35% of Amazon.com’s revenue is generated by its recommendation engine — so what’s their strategy?
- They have billions of data points to test so can try different combinations to quickly find out works and what doesn’t (Artificial Intelligence).
- Amazon does recommendations both on-site and off, via email — guess which performs better? (e-mail!).
- Recommendations can be personalised or based on mass purchases across customer database. Frequently Bought Together Recommender Systems
- Is there any bias in the purchasing process? (Week 7 throwback!)
- Where have Apple tried to up-sell, if at all?
- Where have Apple tried to cross-sell, if at all?
- How much are we paying at checkout? 5. How do they know what to recommend? Let’s Go Shopping INFS 2036 16
O N L I N E R E TA I L E R WA N T I N G
T O I N C R E A S E S A L E S T H R O U G H
C R O S S - + U P - S E L L I N G
Market Basket Analysis (MBA) INFS 2036 19
U S I N G D ATA M I N I N G T O A N T I C I PAT E C U S T O M E R B E H A V I O U R
- MBA is also known as Association Rule Mining and can apply to any business, not just supermarkets.
- Theory: if you buy a certain group of items, you are more (or less) likely to buy another group of items.
- Association Rule: e.g. IF you are in a bar and buy beer and don’t buy a meal THEN you are more likely
to buy a bar snack (chips, pretzels, peanuts …).
IF {beer, no meal} THEN {bar snack}
Antecedent Consequent Itemset = {beer, no meal, bar snack}
IF {this} THEN {that}
general form of an Association Rule DESCRIPTIVE ANALYTICS
Market Basket Analysis INFS 2036 20
H O W D O W E G E T A S S O C I AT I O N R U L E S?
Choose Strong Association Rules Look for Associations Look for items that co-occur Data from POS transactions
- Get data! E.g. a supermarket could use point-of-sale (POS) transaction data or invoices including a list of products purchased, unit price and quantity of each item. To be statistically significant, the data set must be large (e.g. 32 million records of 50,000 unique items).
- Look for and count combinations of items that co-occur frequently otherwise there are too many combinations to work with.
- Look for associations (Market Basket Analysis aka Association Rule Mining). The goal is to find associations that co-occur far more often than in a random sampling of possibilities. We can use Support, Confidence, Lift.
- Choose strong association rules + seek explanations as to why such associations exist and drive business decisions. DESCRIPTIVE ANALYTICS
Market Basket Analysis (MBA) INFS 2036 21
S U P P O R T, C O N F I D E N C E , L I F T
- Support: Itemset popularity calculated as a proportion. E.g. What proportion of purchases contain an apple? P(apple)
- Confidence: conditional probability capturing how likely Y is to be purchased when X is purchased {X - > Y}. E.g. IF {apple} THEN {beer} i.e. also written as {apple} - > {beer} and calculated as P(beer | apple) DESCRIPTIVE ANALYTICS Consequent Antecedent Choose Strong Association Rules Associations^ Look for Look for items that co-occur transactions^ Data from^ POS
Market Basket Analysis INFS 2036 22
S U P P O R T, C O N F I D E N C E , L I F T
DESCRIPTIVE ANALYTICS
- Confidence might misrepresent the importance of an association as it only accounts for how popular apples are, but not beers. If beers are also very popular in general, there will be a higher chance that a transaction containing apples will also contain beers, thus inflating the confidence measure.
- What level of Confidence (high or low) would you give for the following Association Rules?
- {Butter} →{Bread}
- {Cereal} → {Milk}
- {Flour, Sugar} → {Milk}
- {Bread, Egg} → {Milk}
- {Toothbrush} → {Milk} Use Lift to overcome this problem. THEORY PRACTICE Choose Strong Association Rules Associations^ Look for Look for items that co-occur transactions^ Data from^ POS
Market Basket Analysis INFS 2036 25
S U P P O R T, C O N F I D E N C E , L I F T
What do these associations tell us?
- {beer - > soda}: has the highest confidence (20%). However both items have high support so is the combination random or a genuine combination? Lift = 1 implies no association between beer and soda.
- {beer - > berries}: beer has high support and berries has low support, but now Lift = 0.3. This suggests that if someone buys berries they would likely not buy beer.
- {beer - > male cosmetics}: has low confidence due to low support for Male Cosmetics even though beer has high support. However Lift = 2.6 implying when someone does buy Male Cosmetics they are very likely to buy beer as well. THEORY PRACTICE DESCRIPTIVE ANALYTICS Choose Strong Association Rules Associations^ Look for Look for items that co-occur transactions^ Data from^ POS
- A data set of over 21,000 transactions in a Spanish bakery was recorded.
- After cleaning for repeated purchases, removing erroneous data and
data that looked unreliable, a refined data set of 6614 transactions over
104 items was available for Market Basket Analysis.
- What does the data look like? ( descriptive data mining)
- Which will be the best-selling items? ( predictive data mining) Market Basket Analysis INFS 2036 26
E X A M P L E : B A K E R Y T R A N S A C T I O N S
DESCRIPTIVE ANALYTICS PREDICTIVE ANALYTICS THEORY PRACTICE
Market Basket Analysis INFS 2036 27
E X A M P L E : B A K E R Y T R A N S A C T I O N S
THEORY PRACTICE DESCRIPTIVE ANALYTICS
Market Basket Analysis INFS 2036 28
E X A M P L E : B A K E R Y T R A N S A C T I O N S
THEORY PRACTICE Support, Confidence
- Lift were calculated for each association as well as frequency (count). This table can be also visualised as a graph. DESCRIPTIVE ANALYTICS PREDICTIVE ANALYTICS
INFS 2036 31 In practice, a rule needs the support of several hundred transactions, before it can be considered statistically significant, and data sets often contain thousands or millions of transactions. Automation of association rules — Artificial Intelligence — is used widely for this reason!
INFS 2036 32 Market Basket Analysis B R O A D E R R A N G E O F P R A C T I C A L A P P L I C AT I O N S There are many areas in which Market Basket Analysis can be applied, including:
- Retail: what items are purchased together, sequentially and by season. Assists product placement or promotion to cross- and up-sell or make recommendations. E.g. chips + soft drink or chips + imported beer?
- Telecommunications: services and packages customers would purchase. E.g. bundling TV + Internet. This gives insight into pricing bundles and determining capacity requirements.
- Banking: analyse credit card purchases to build profiles for fraud detection and cross-selling opportunities (e.g. insurance, wine, holidays … ).
- Insurance: build profiles to detect insurance claim fraud to determine if more than 1 claim belongs to a particular claimee within a specified period of time.
- Medical: identify co-morbid conditions and symptom analysis to better identify an illness profile.
Market Basket Analysis INFS 2036 33
S U P P O R T, C O N F I D E N C E O R L I F T?
- The most popular transaction was …
- Another popular transaction was …
- If someone buys meat spreads then …
- Many people buy sausage along with …
- Given someone buys tea, they are likely to buy … THEORY PRACTICE Based on the supermarket purchasing patterns, complete the statements below using support, confidence and/or lift where appropriate. Support: circle size. Larger circles imply higher support. Lift: circle colour. Higher lift is red, lower lift is yellow.
ACTIVITY
INFS 2036 34 Focus on Week 8 P R E D I C T I V E D ATA M I N I N G
Model + Predicting Customer Behaviour via Predictive Data Mining
- Predictive data mining supports predictive analytics using current + past data to make future predictions. PREDICTIVE DATA MINING PREDICTIVE ANALYTICS
- The process of going through system databases and finding relevant data to analyse for the purpose of prediction.
- E.g. use algorithm-based tools to go through a customer database to look at past transactions to support theories about possible future volumes of transactions.
- The use of data found by data mining to build statistical forecasting models to predict trends.
- E.g. what will the customer buy next? What will the customer churn rate be? How many new subscribers will we gain from this offer? What amount of stock do we need for this product for the next month?
37 PREDICTIVE DATA MINING A L G O R I T H M S INFS 2036
- What is an algorithm? A sequence of instructions with a clear stopping point to solve a problem.
- Many algorithms in life: e.g. a recipe, driving directions, instructions to construct furniture (IKEA), instructions on how to complete an exam … also for solving business problems.
- E.g. we want to find association rules (models) for a bakery:
- Algorithm: program code (the algorithm) to extract data, check it and analyse it. The program code is the list of instructions (algorithm) that performs all of these functions and more.
- Stopping point: association rules with statistics like confidence, support and lift. rules_sup1_conf50 <- apriori(trans, parameter=list(sup=supportLevels[3], conf=confidenceLevels[5], target="rules")) inspect(rules_sup1_conf50) plot(rules_sup1_conf50,method='graph',control=list(nodeCol=brewer.pal(8,'BuGn’), edgeCol="royalblue2", alpha=1, cex=2, lab="black",arrowSize=.45),shading="lift")
38 PREDICTIVE DATA MINING M O D E L S
- What is a model? An equation or series of equations representing the problem of interest using data to produce an answer to a question (e.g. association rules for a bakery).
- Predictive Models split the available data into two groups for training and then testing, i.e.:
- Training data: used to develop (train) a model that can be used for prediction of different or new data.
- Testing data: data used to test and validate the model’s predictive capability against known results. Used to see how well the model works before using it to forecast without the safety of validation data!
- We consider two types of Regression predictive models:
- Value Prediction: specifically uses linear regression models for prediction of a value. E.g. how much revenue will be generated over a year.
- Classification Models: predict class (category) membership uses logistic and ordinal regression. E.g. good or bad credit risk or which car size will be most popular. INFS 2036
INFS 2036 39 CLASSIFICATION MODELS P R E D I C T I V E D ATA M I N I N G TWO-CLASS Strictly only two answers
- Yes/No
- True/False
- Good/Bad
- Concession/No Concession
- Renewed/Not Renewed
- Tweet sentiment is positive/negative
- … MULTI-CLASS More than two answers
- The tweet sentiment is positive/negative/neutral
- Sunny, Cloudy, Rainy, Snow
- How does a customer feel about the discount?
- What does a customer think about the promotion?
- Which promotions are more popular?
- … Logistic Regression Ordinal Regression
Steps 1 + 2: Get the data + Prepare the data A good data warehouse supports predictive data mining (Week 9).
- Data Standards have been applied
- Data Quality is assured
- Metadata to help us choose which data to use for prediction (e.g. time of day, weather, holiday deals etc). Step 3: Train the model Use part of the data to create a model e.g. car preference — logistic/ordinal regression (categories) e.g. car demand — linear regression (values) Step 4: Evaluate the model Use the model to predict the test data set and compare the predictions to the known output in the test data set. Step 5: Predict! INFS 2036 Use the model to predict demand + hope you did a good job^ J 40 PREDICTIVE DATA MINING B I G P I C T U R E S T E P S T O P R O D U C E A M O D E L THEORY PRACTICE PREDICTIVE DATA MINING PREDICTIVE ANALYTICS
