Download CNLR - Mathematics and Statistics - Study Notes and more Study notes Mathematical Statistics in PDF only on Docsity!
1
CNLR is used to estimate the parameters of a function by minimizing a smooth nonlinear loss function (objective function) when the parameters are subject to a set of constraints.
Model
Consider the model
f = f (x ~ (^) , θ ~ )
where θ ~ is a p × 1 parameter vector, x~ is an independent variable vector, and f is a function of x~ and θ~.
Goal
Find the estimate θ ~ ∗of θ ~ such that θ ~ ∗^ minimizes
F = F y f 1 , 6
subject to
l A q C
~ u
~ ≤ ~
% &
K
' K
( )
K
K
θ
θ
L ( (^) ~)
~
where F is the smooth loss function (objective function), which can be specified by the user. A (^) L is an m (^) L × p matrix of linear constraints, and C ( θ~ ) is an m (^) N × 1 vector of nonlinear constraint functions. (^) l ~ (^) ′^ = ( l ~ (^) ′ B (^) , l ~ (^) ′ L (^) , l ~′ N ), where (^) l ~ B ′ , (^) l ~ ′ L ,
and (^) l ~ (^) N^ ′^ represent the lower bounds, linear constraints and nonlinear constraints, respectively. The upper bound u~ is defined similarly.
Algorithm
CNLR uses the algorithms proposed and implemented in NPSOL by Gill, Murray, Saunders, and Wright. A description of the algorithms can be found in the User’s Guide for NPSOL , Version 4.0 (1986). The method used in NPSOL is a sequential quadratic programming (SQP) method. For an overview of SQP methods, see Gill, Murray, and Wright (1981), pp. 237–242. The basic structure of NPSOL involves major and minor iterations. Based on the given initial value θ~( )^0 of θ~ , the algorithm first selects an initial working set that includes bounds or general inequality constraints that lie within a crash tolerance (CRSHTOL). At the k th iteration, the algorithm starts with
(I) Minor Iteration
This iteration searches for the direction Pk , which is the solution of a quadratic subproblem; that is, Pk , is found by minimizing
g P ′ k (^) + 21 P H P ′ k (1)
subject to
l
P
A P
A P
~(^^ k^ ) L u ~(^ ) N
≤ k
% &
K
'
K
( )
K
K
where g k is the gradient of F at θ ~^ (^ k^ ), the matrixp H k is a positive-definite quasi-
Newton approximation to the Hessian of the Lagrangian function, A (^) N is the Jacobian matrix of the nonlinear-constraint vector C evaluated at θ~(^ k^ ), and
satisfies the following conditions:
- (^) θ~^1^ k^^ +^16 produces a “sufficient decrease” in the augmented Lagrangian merit
function
L F (^) i ci s (^) i c s i
i i i i
( ,θ λ~ ~ , ) s~ = ( )~θ − ( )θ ~ − ( )~θ
∑ ∑ λ ρ
2 (2)
The summation terms in (2) involve only the nonlinear constraints. The vector λ is an estimate of the Lagrange multipliers for the nonlinear constraints. The non-negative slack variables (^) ; @ si allow nonlinear inequality constraints to be
treated without introducing discontinuities. The solution of the QP subproblem defined in (1) provides a vector triple that serves as a direction search for θ~ ,
λ~ and s~. The non-negative vector of penalty parameters1 6 ρ is initialized to
zero at the beginning of the first major iteration. Function precision criteria are used as a measure of the accuracy with which the functions F and ci can be
evaluated.
- θ~^1^ k^^ +^16 is close to a minimum of F along P k. The criterion is
g ′ ( (^) ~ θ 1 k^ +^16 ) P k (^) < − η g P ′ k k
where η is the Line Search Tolerance and 0 ≤ η < 1. The value of η determines the accuracy with which α (^) k approximates a stationary point of F along Pk. A smaller value of^ η^ produces a more accurate line search.
- The step length is in a certain range; that is,
θ~ 1^ k^^ +^16 − θ~1 6 k^ = α (^) k P k ≤Step Limit.
(III) Convergence Tests
After α (^) k is determined from the major iteration, the following conditions are checked:
- k + 1 ≤Maximum number of major iterations
- The sequence {θ (^) ~ 1 6 l^ }converged at θ ~^1^ k^ +^16 ; that is,
α (^) k P k (^) ≤ r^ + k
1 || +^1 ||
~^ θ
1 6
- θ~^1^ k^^ +^16 satisfies the Kuhn-Tucker conditions to the accuracy requested; that is,
|| g (^) z ( θ ~ k^ +^ )|| ≤ r + max +| F ( (^) ~θ k^ +^ )|,|| g ( θ~ k + )||
^
1 1 6 1 1 1 16 1 16
and
res (^) j ≤ FTOL, for all j ,
where g z is the projected gradient, g is the gradient of F with respect to the free parameters, res (^) j is the violation of the j th nonlinear constraint, FTOL is the Nonlinear Feasibility Tolerance, and r is the Optimality Tolerance.
If none of these three conditions are satisfied, the algorithm continues with the Minor Iteration to find a new search direction.
Termination
The following are termination conditions.
- Underflow. A single underflow will always occur if machine constants are computed automatically. Other floating-point underflows may occur occasionally, but can usually be ignored.
- If the loss function is not specified, the default loss function is a squared loss function and the default output in NLR will be printed. However, if the loss function is not a squared loss function, CNLR prints only the final parameter estimates, iteration history, and termination message. In order to obtain estimates of the standard errors of parameter estimates and correlations between parameter estimates, the bootstrapping method can be requested.
Bootstrapping Estimates
Bootstrapping is a nonparametric technique of estimating the standard error of a parameter estimate using repeated samples from the original data. This is done by sampling with replacement. CNLR computes and saves the parameter estimates for each sample generated. This results, for each parameter, in a sample of estimates from which the standard deviation is calculated. This is the estimated standard error. Mathematically, the bootstrap covariance matrix S for the p parameter estimates is
S =
× sij 3 8 p p
where
s
m
ij ik i jk j k
m
i ik k
m
=
=
∑
∑
θ θ θ θ
θ θ
4 94 9 1
1
and θ$ ik is the CNLR parameter estimate of θ (^) i for the k th bootstrap sample and m
is the number of samples generated by the bootstrap. By default, m
p p
1 6 .
The standard error for the j th parameter estimate is estimated by
s m
jj − 1
and the correlation between the i th and j th parameter estimates is estimated by
s s s
ij ii jj
The “95% Trimmed Range” values are the most extreme values that remain after trimming from the set of estimates for a parameter, the g largest, and the g smallest estimates, where g is the largest integer not exceeding 0.025 m.
References
Gill, P. E., Murray, W. M., and Wright, M. H. 1981. Practical Optimization. London: Academic Press.
Gill, P. E., Murray, W. M., Saunders, M. A., and Wright, M. H. 1984. Procedures for optimization problems with a mixture of bounds and general linear constraints. ACM Transactions on Mathematical Software , 10 (3): 282–296.
Gill, P. E., Murray, W. M., Saunders, M. A., and Wright, M. H. 1986. User’s guide for NPSOL ( version 4.0): A fortran package for nonlinear programming. Technical Report SOL 86–2, Department of Operations Research, Stanford University.
