Linear Optimization Lecture 10: Regression and Optimization with Means and Medians, Study notes of Production and Operations Management

Download Linear Optimization Lecture 10: Regression and Optimization with Means and Medians and more Study notes Production and Operations Management in PDF only on Docsity!

OPIM 913

Linear Optimization

Lecture 10

Regression

• Means and Medians
• Least Squares Regression
• Least Absolute Deviation (LAD) Regression
• LAD via LP
• Average Complexity of Primal-Dual Simplex Method

1

Means and Medians

Consider 1995 Adjusted Gross Incomes on Individual Tax Returns:

b 1 b 2 b 3... b (^) m − 1 b (^) m

$25,462 $45,110 $15,505... $33,265 $75,

Median:

x ˆ = b 1 + m 2

Mean:

x ¯ =

m

m ∑

i = 1

bi = $4,189,353,615,000/118,218,327 = $35,437.

Median’s Connection with Optimization

x ˆ = argmin x

m ∑

i = 1

| x − bi |.

Proof:

f ( x ) =

m ∑

i = 1

| x − bi |

f

′ ( x ) =

m ∑

i = 1

sgn (^) ( x − bi ) where sgn( x ) =

1 x > 0

0 x = 0

− 1 x < 0

= (# of b (^) i ’s smaller than x ) − (# of b (^) i ’s larger than x ).

If m is odd:

1

3

5

−

−

−

A Regression Model for Algorithm Efficiency

t = # of iterations

m = # of constraints

n = # of variables

Model:

t ≈ 2

α ( m + n )

β

Linearization: Take logs:

log t = α log 2 + β log( m + n ) + 

↑

error (difference w/ model prediction)

Solve several (k) instances: 

log t 1

log t 2 .. .

log t (^) k

log 2 log( m 1 + n 1 )

log 2 log( m 2 + n 2 ) .. .

log 2 log( m (^) k + n (^) k )

[

α

β

]

 k

In matrix notation:

b = Ax + 

Goal: find x that “minimizes” .

Least Absolute Deviation Regression

Definition: Manhattan Distance

‖ x ‖ 1 =

i

| x (^) i |

Least Absolute Deviation Regression:

x ˆ = argmin x ‖ b − Ax ‖ 1

Calculus (no explicit formula this time but iterative scheme):

f ( x ) = ‖ b − Ax ‖ 1 =

i

bi −

j

ai j x (^) j

∂ f

∂ x (^) k

( x ˆ) =

i

bi −

j ai j x ˆ (^) j ∣ ∣ ∣ bi −^

j ai j x ˆ (^) j

(− aik ) = 0 , k = 1 , 2 ,... , n

Rearranging,

i

aik bi

 i ( x ˆ)

i

j

aik ai j x ˆ (^) j

( x ˆ)

, k = 1 , 2 ,... , n

In matrix notation,

A

T E ( x ˆ) b = A

T E ( x ˆ) A x ˆ

where

E ( x ˆ) = Diag(( x ˆ))

− 1

Assuming A

T E ( x ˆ) A is invertible,

x ˆ =

A

T E ( x ˆ ) A

A

T E ( x ˆ ) b, where E depends on ˆx

Least Absolute Deviation Regression—Continued

An implicit equation.

Can be solved using successive approximations:

x

0 = 0

x

1

A

T E ( x

0 ) A

A

T E ( x

0 ) b

x

2

A

T E ( x

1 ) A

A

T E ( x

1 ) b

.. .

x

k + 1

A

T E ( x

k ) A

A

T E ( x

k ) b

.. .

x ˆ = lim k →∞

x

k

Primal–Dual Simplex Method

Thought experiment:

• μ starts at ∞.
• In reducing μ, there are n + m barriers.
• At each iteration, one barrier is passed—the others move about randomly.
• To get μ to zero, we must on average pass half the barriers.
• Therefore, on average the algorithm should take ( m + n )/2 iterations.

Least Squares Regression:

[ α¯

β^ ¯

]

[

]

⇒ T ≈ 0. 488 ( m + n )

1. 052

Least Absolute Deviation Regression:

[ αˆ

β^ ˆ

]

[

]

⇒ T ≈ 0. 517 ( m + n )

1. 049

Primal–Dual Simplex Method: Data

• 25fv47 777 1545 5089 nesm Name m n iters Name m n iters
• 80bau3b 2021 9195 10514 recipe
• adlittle 53 96 141 sc105
• afiro 25 32 16 sc205
• agg2 481 301 204 sc50a
• agg3 481 301 193 sc50b
• bandm 224 379 1139 scagr25
• beaconfd 111 172 113 scagr7
• blend 72 83 117 scfxm1
• bnl1 564 1113 2580 scfxm2
• bnl2 1874 3134 6381 scfxm3
• boeing1 298 373 619 scorpion
• boeing2 125 143 168 scrs8
• bore3d 138 188 227 scsd1
• brandy 123 205 585 scsd6
• czprob 689 2770 2635 scsd8
• d6cube 403 6183 5883 sctap1
• degen2 444 534 1421 sctap2
• degen3 1503 1818 6398 sctap3
• e226 162 260 598 seba
• etamacro 334 542 1580 share1b
• fffff800 476 817 1029 share2b
• finnis 398 541 680 shell
• fit1d 24 1026 925 ship04l
• fit1p 627 1677 15284 ship04s
• forplan 133 415 576 ship08l
• ganges 1121 1493 2716 ship08s
• greenbea 1948 4131 21476 ship12l
• grow15 300 645 681 ship12s
• grow22 440 946 999 sierra
• grow7 140 301 322 standata
• israel 163 142 209 standmps
• kb2 43 41 63 stocfor1
• lotfi 134 300 242 stocfor2
• maros