Linear Programming Regression, Lecture Notes - Mathematics, Study notes of Linear Programming

Download Linear Programming Regression, Lecture Notes - Mathematics and more Study notes Linear Programming in PDF only on Docsity!

Linear Programming: Chapter 12

Regression

Robert J. Vanderbei

October 17, 2007

Operations Research and Financial Engineering Princeton University Princeton, NJ 08544 http://www.princeton.edu/∼rvdb

Outline

• Means and Medians
• Least Squares Regression
• Least Absolute Deviation (LAD) Regression
• LAD via LP
• Average Complexity of Parametric Self-Dual Simplex Method

Means and Medians

Median: x ˆ = b1+ 2 m ≈ $22,500.

Mean:

x ¯ =

m

∑^ m

i=

bi = $4,189,353,615,000/118,218,327 = $35,437.

Median’s Connection with Optimization

x ˆ = argminx

∑m

i=

|x − bi|.

Proof:

f (x) =

∑^ m

i=

|x − bi|

f ′(x) =

∑^ m

i=

sgn (x − bi) where sgn(x) =

1 x > 0 0 x = 0 − 1 x < 0 = (# of bi’s smaller than x) − (# of bi’s larger than x).

If m is odd:

1

3

5

− − −

A Regression Model for Algorithm Efficiency

Observed Data:

t = # of iterations m = # of constraints n = # of variables

Model: t ≈ 2 α(m + n)β

Linearization: Take logs:

log t = α log 2 + β log(m + n) +  ↑ error

Least Squares Regression

Euclidean Distance: ‖x‖ 2 = (∑ i x^2 i^ )^1 /^2 Least Squares Regression: x¯ = argminx‖b − Ax‖^22 Calculus:

f (x) = ‖b − Ax‖^22 =

∑ i

 bi − ∑ j

aij xj

 

2

∂f ∂xk^ (¯x) =^

∑ i

2

 bi − ∑ j

aij ¯xj

  (^) (−aik) = 0, k = 1, 2 ,... , n

Rearranging, (^) ∑

i

aikbi =

∑ i

∑ j

aikaij ¯xj , k = 1, 2 ,... , n

In matrix notation, AT^ b = AT^ Ax¯ Assuming AT^ A is invertible, ¯x =

( AT^ A

)− 1 AT^ b

Least Absolute Deviation Regression

Manhattan Distance: ‖x‖ 1 = ∑ i |xi| Least Absolute Deviation Regression: xˆ = argminx‖b − Ax‖ 1 Calculus:

f (x) = ‖b − Ax‖ 1 =

∑ i

∣∣ ∣∣ ∣∣bi^ −^

∑ j

aij xj

∣∣ ∣∣ ∣∣

∂f ∂xk^ (ˆx) =^

∑ i

∣^ bi^ −^ ∑^ j^ aij^ xˆj ∣∣bi − ∑ j aij^ xˆj

∣∣ ∣

(−aik) = 0, k = 1, 2 ,... , n

Rearranging, (^) ∑

i

aikbi i(ˆx) =^

∑ i

∑ j

aikaij xˆj i(ˆx) ,^ k^ = 1,^2 ,... , n

In matrix notation, AT^ E(ˆx)b = AT^ E(ˆx)Ax,ˆ where E(ˆx) = Diag((ˆx))−^1

Assuming AT^ E(ˆx)A is invertible, ˆx =

( AT^ E(ˆx)A

)− 1 AT^ E(ˆx)b

Least Absolute Deviation Regression via Lin-

ear Programming

min

i

bi −

j

aij xj

Equivalent Linear Program:

min

i

ti

−ti ≤ bi −

j

aij xj ≤ ti i = 1, 2 ,... , m

AMPL Model

param m; param n;

set I := {1..m}; set J := {1..n};

param A {I,J}; param b {I};

var x{J}; var t{I};

minimize sum_dev: sum {i in I} t[i];

subject to lower_bound {i in I}: -t[i] <= b[i] - sum {j in J} A[i,j]*x[j];

subject to upper_bound {i in I}: b[i] - sum {j in J} A[i,j]*x[j] <= t[i];

Parametric Self-Dual Simplex Method: Data

Data Continued

• 25fv47 777 1545 5089 nesm Name m n iters Name m n iters
• 80bau3b 2021 9195 10514 recipe
• adlittle 53 96 141 sc105
• afiro 25 32 16 sc205
• agg2 481 301 204 sc50a
• agg3 481 301 193 sc50b
• bandm 224 379 1139 scagr25
• beaconfd 111 172 113 scagr7
• blend 72 83 117 scfxm1
• bnl1 564 1113 2580 scfxm2
• bnl2 1874 3134 6381 scfxm3
• boeing1 298 373 619 scorpion
• boeing2 125 143 168 scrs8
• bore3d 138 188 227 scsd1
• brandy 123 205 585 scsd6
• czprob 689 2770 2635 scsd8
• d6cube 403 6183 5883 sctap1
• degen2 444 534 1421 sctap2
• degen3 1503 1818 6398 sctap3
• e226 162 260 598 seba
• etamacro 334 542 1580 share1b Name m n iters Name m n iters
• fffff800 476 817 1029 share2b
• finnis 398 541 680 shell
• fit1d 24 1026 925 ship04l
• fit1p 627 1677 15284 ship04s
• forplan 133 415 576 ship08l
• ganges 1121 1493 2716 ship08s
• greenbea 1948 4131 21476 ship12l
• grow15 300 645 681 ship12s
• grow22 440 946 999 sierra
• grow7 140 301 322 standata
• israel 163 142 209 standmps
• kb2 43 41 63 stocfor1
• lotfi 134 300 242 stocfor2
• maros