Data Mining & Machine Learning: Quiz on Search Procedures and Optimization, Lecture notes of Artificial Intelligence

Download Data Mining & Machine Learning: Quiz on Search Procedures and Optimization and more Lecture notes Artificial Intelligence in PDF only on Docsity!

Data Mining & Machine Learning

CS

Purdue University September 13, 2022

Quiz

• Naive Bayes Classifier challenges:
  • (a) Over-counting evidence, leads to overly confident probability estimate
  • (b) Zero counts are a problem because it affects the conditional probability and has no known solution

Combinatorial optimization

Optimization

• Non-smooth functions:
  • If the function is discrete , then traditional optimization methods that rely on smoothness are not applicable (e.g., gradient descent needs the derivative). Instead we need to use combinatorial optimization - Example: Choosing what features (structure) to add to a decision tree

Heuristic search

• Typically, there is an exponential number of models in the (discrete) search space, making it intractable to exhaustively search the space - Thus, it is generally impossible to return a model that is guaranteed to have the best score
• Instead, we have to resort to heuristic search techniques
  • Methods are evaluated experimentally and shown to have good performance on average
  • Greedy search: Given a current model M, look for other models near M and move to the best of these (if any have a score better than M)

Greedy search

• Choose an initial state M^0 corresponding to a particular model structure (e.g., an empty tree)
• Let Mi^ be the model structure location at the i-th iteration
• For each iteration i
  • Construct all possible models {Mj1, ..., Mjk} adjacent to Mi^ (as defined by search operators)
  • Evaluate scores for all models {Mj1, ..., Mjk}
  • Choose to move to the adjacent model with best score: Mi+1^ = Mj.best
  • Repeat until there is no possible further improvement in the score

Greedy tree search algorithm

• Input:
  • Initial state? State space?
  • Set of actions?
  • How to choose next state?
  • Goal test?
• Output: -?

Root feature

Root feature, Root feature,

• X
• X
• X
  • X1,X
  • X2,X
  • X3,X
  • X1,X left child
  • X2,X
  • X3,X
    • X1,X2,X
    • X2,X3,X
    • X3,X1,X
    • X1,X3,X
    • X2,X1,X
    • X3,X2,X
    • X1,X2,X left child, right child
    • X1,X3,X
    • X2,X1,X
    • X2,X3,X
    • X3,X1,X
    • X3,X2,X

Questions to ask about search procedures

• Is the search exhaustive?
  • I.e., does it either explicitly or implicitly consider all models in the space?
• Is the search optimal?
  • I.e., is it guaranteed to return the model with the best score?
  • Global vs. local optimum?

Smooth optimization

Convex optimization problems

• Where f is a convex score function C is a convex set ( constraints on model parameters or structure ) are the parameters variable and x is the data
• For convex optimization problems, all locally optimal points are globally optimal
• Example algorithms: Quadratic programming (SVMs), least squares estimation, maximum likelihood estimation

minimize f ( x ; θ )

subject to θ ∈ C

Convex optimization

Convex functions

• In graph of convex function, the line connecting two points must lie above the function f (↵x + (1 ↵)y)  ↵f (x) + (1 ↵)f (y) for all 0  ↵  1 A function f is convex if:

Concave vs convex

• Maximizing a concave function is equivalent to minimizing a convex function