ó î&]\c@`sÉdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddd gZ defd „ƒYZd efd „ƒYZdefd „ƒYZd efd„ƒYZdS(s@Hessian update strategies for quasi-Newton optimization methods.i(tdivisiontprint_functiontabsolute_importN(tnorm(tget_blas_funcs(twarntHessianUpdateStrategytBFGStSR1cB`s2eZdZd„Zd„Zd„Zd„ZRS(s]Interface for implementing Hessian update strategies. Many optimization methods make use of Hessian (or inverse Hessian) approximations, such as the quasi-Newton methods BFGS, SR1, L-BFGS. Some of these approximations, however, do not actually need to store the entire matrix or can compute the internal matrix product with a given vector in a very efficiently manner. This class serves as an abstract interface between the optimization algorithm and the quasi-Newton update strategies, giving freedom of implementation to store and update the internal matrix as efficiently as possible. Different choices of initialization and update procedure will result in different quasi-Newton strategies. Four methods should be implemented in derived classes: ``initialize``, ``update``, ``dot`` and ``get_matrix``. Notes ----- Any instance of a class that implements this interface, can be accepted by the method ``minimize`` and used by the compatible solvers to approximate the Hessian (or inverse Hessian) used by the optimization algorithms. cC`stdƒ‚dS(sÏInitialize internal matrix. Allocate internal memory for storing and updating the Hessian or its inverse. Parameters ---------- n : int Problem dimension. approx_type : {'hess', 'inv_hess'} Selects either the Hessian or the inverse Hessian. When set to 'hess' the Hessian will be stored and updated. When set to 'inv_hess' its inverse will be used instead. s=The method ``initialize(n, approx_type)`` is not implemented.N(tNotImplementedError(tselftnt approx_type((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt initialize%scC`stdƒ‚dS(söUpdate internal matrix. Update Hessian matrix or its inverse (depending on how 'approx_type' is defined) using information about the last evaluated points. Parameters ---------- delta_x : ndarray The difference between two points the gradient function have been evaluated at: ``delta_x = x2 - x1``. delta_grad : ndarray The difference between the gradients: ``delta_grad = grad(x2) - grad(x1)``. s>The method ``update(delta_x, delta_grad)`` is not implemented.N(R (R tdelta_xt delta_grad((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pytupdate7scC`stdƒ‚dS(sQCompute the product of the internal matrix with the given vector. Parameters ---------- p : array_like 1-d array representing a vector. Returns ------- Hp : array 1-d represents the result of multiplying the approximation matrix by vector p. s)The method ``dot(p)`` is not implemented.N(R (R tp((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pytdotIscC`stdƒ‚dS(söReturn current internal matrix. Returns ------- H : ndarray, shape (n, n) Dense matrix containing either the Hessian or its inverse (depending on how 'approx_type' is defined). s0The method ``get_matrix(p)`` is not implemented.N(R (R ((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt get_matrixZs (t__name__t __module__t__doc__R RRR(((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR s    tFullHessianUpdateStrategycB`s†eZdZedddƒZedddƒZedddƒZdd„Zd„Zd „Z d „Z d „Z d „Z d „Z RS(sKHessian update strategy with full dimensional internal representation. tsyrtdtypetdtsyr2tsymvtautocC`s1||_d|_d|_d|_d|_dS(N(t init_scaletNonetfirst_iterationR tBtH(R R((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt__init__ps     cC`s|t|_||_||_|dkr6tdƒ‚n|jdkr`tj|dtƒ|_ntj|dtƒ|_ dS(sÏInitialize internal matrix. Allocate internal memory for storing and updating the Hessian or its inverse. Parameters ---------- n : int Problem dimension. approx_type : {'hess', 'inv_hess'} Selects either the Hessian or the inverse Hessian. When set to 'hess' the Hessian will be stored and updated. When set to 'inv_hess' its inverse will be used instead. thesstinv_hesss+`approx_type` must be 'hess' or 'inv_hess'.RN(R$R%( tTrueR R R t ValueErrortnpteyetfloatR!R"(R R R ((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR ys    cC`sŠtj||ƒ}tj||ƒ}tjtj||ƒƒ}|dksc|dksc|dkrgdS|jdkr~||S||SdS(NgiiR$(R(RtabsR (R RRts_norm2ty_norm2tys((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt _auto_scale“s$cC`stdƒ‚dS(Ns9The method ``_update_implementation`` is not implemented.(R (R RR((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt_update_implementation¡scC`sËtj|dkƒrdStj|dkƒr?tdtƒdS|jr·|jdkrl|j||ƒ}nt|jƒ}|jdkrœ|j |9_ n|j |9_ t |_n|j ||ƒdS(söUpdate internal matrix. Update Hessian matrix or its inverse (depending on how 'approx_type' is defined) using information about the last evaluated points. Parameters ---------- delta_x : ndarray The difference between two points the gradient function have been evaluated at: ``delta_x = x2 - x1``. delta_grad : ndarray The difference between the gradients: ``delta_grad = grad(x2) - grad(x1)``. gNsÇdelta_grad == 0.0. Check if the approximated function is linear. If the function is linear better results can be obtained by defining the Hessian as zero instead of using quasi-Newton approximations.RR$( R(tallRt UserWarningR RR/R*R R!R"tFalseR0(R RRtscale((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR¥s  cC`s?|jdkr%|jd|j|ƒS|jd|j|ƒSdS(sQCompute the product of the internal matrix with the given vector. Parameters ---------- p : array_like 1-d array representing a vector. Returns ------- Hp : array 1-d represents the result of multiplying the approximation matrix by vector p. R$iN(R t_symvR!R"(R R((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyRËscC`s`|jdkr$tj|jƒ}ntj|jƒ}tj|ddƒ}|j|||<|S(sïReturn the current internal matrix. Returns ------- M : ndarray, shape (n, n) Dense matrix containing either the Hessian or its inverse (depending on how `approx_type` was defined). R$tkiÿÿÿÿ(R R(tcopyR!R"ttril_indices_fromtT(R tMtli((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyRÞs (RRRRt_syrt_syr2R5R#R R/R0RRR(((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyRhs    & cB`s;eZdZdddd„Zd„Zd„Zd„ZRS(sÒBroyden-Fletcher-Goldfarb-Shanno (BFGS) Hessian update strategy. Parameters ---------- exception_strategy : {'skip_update', 'damp_update'}, optional Define how to proceed when the curvature condition is violated. Set it to 'skip_update' to just skip the update. Or, alternatively, set it to 'damp_update' to interpolate between the actual BFGS result and the unmodified matrix. Both exceptions strategies are explained in [1]_, p.536-537. min_curvature : float This number, scaled by a normalization factor, defines the minimum curvature ``dot(delta_grad, delta_x)`` allowed to go unaffected by the exception strategy. By default is equal to 1e-8 when ``exception_strategy = 'skip_update'`` and equal to 0.2 when ``exception_strategy = 'damp_update'``. init_scale : {float, 'auto'} Matrix scale at first iteration. At the first iteration the Hessian matrix or its inverse will be initialized with ``init_scale*np.eye(n)``, where ``n`` is the problem dimension. Set it to 'auto' in order to use an automatic heuristic for choosing the initial scale. The heuristic is described in [1]_, p.143. By default uses 'auto'. Notes ----- The update is based on the description in [1]_, p.140. References ---------- .. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization" Second Edition (2006). t skip_updateRcC`s|dkr0|dk r$||_qld|_n<|dkr`|dk rT||_qld|_n tdƒ‚tt|ƒj|ƒ||_dS(NR>g:Œ0âŽyE>t damp_updategš™™™™™É?s<`exception_strategy` must be 'skip_update' or 'damp_update'.(Rt min_curvatureR'tsuperRR#texception_strategy(R RBR@R((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR#s         cC`sS|jd|||d|jƒ|_|j|||d|d|jƒ|_dS(sUpdate the inverse Hessian matrix. BFGS update using the formula: ``H <- H + ((H*y).T*y + s.T*y)/(s.T*y)^2 * (s*s.T) - 1/(s.T*y) * ((H*y)*s.T + s*(H*y).T)`` where ``s = delta_x`` and ``y = delta_grad``. This formula is equivalent to (6.17) in [1]_ written in a more efficient way for implementation. References ---------- .. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization" Second Edition (2006). gð¿taiN(R=R"R<(R R.tHytyHyts((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt_update_inverse_hessian&s%cC`sH|jd||d|jƒ|_|jd||d|jƒ|_dS(sœUpdate the Hessian matrix. BFGS update using the formula: ``B <- B - (B*s)*(B*s).T/s.T*(B*s) + y*y^T/s.T*y`` where ``s`` is short for ``delta_x`` and ``y`` is short for ``delta_grad``. Formula (6.19) in [1]_. References ---------- .. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization" Second Edition (2006). gð?RCgð¿N(R<R!(R R.tBstsBsty((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyt_update_hessian:s"c C`s§|jdkr|}|}n |}|}tj||ƒ}|j|ƒ}|j|ƒ}|dkré|j||ƒ}|jdkr©|tj|jdtƒ|_n|tj|jdtƒ|_|j|ƒ}|j|ƒ}n||j |kre|j dkrdS|j dkred|j d||} | |d| |}tj||ƒ}qen|jdkr|j ||||ƒn|j ||||ƒdS(NR$gRR>R?i( R R(RR/R)R R*R!R"R@RBRKRG( R RRtwtztwztMwtwMwR4t update_factor((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR0Ls2  "N(RRRRR#RGRKR0(((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyRðs !   cB`s&eZdZddd„Zd„ZRS(sÏSymmetric-rank-1 Hessian update strategy. Parameters ---------- min_denominator : float This number, scaled by a normalization factor, defines the minimum denominator magnitude allowed in the update. When the condition is violated we skip the update. By default uses ``1e-8``. init_scale : {float, 'auto'}, optional Matrix scale at first iteration. At the first iteration the Hessian matrix or its inverse will be initialized with ``init_scale*np.eye(n)``, where ``n`` is the problem dimension. Set it to 'auto' in order to use an automatic heuristic for choosing the initial scale. The heuristic is described in [1]_, p.143. By default uses 'auto'. Notes ----- The update is based on the description in [1]_, p.144-146. References ---------- .. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization" Second Edition (2006). g:Œ0âŽyE>RcC`s#||_tt|ƒj|ƒdS(N(tmin_denominatorRARR#(R RRR((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR#–s cC`sß|jdkr|}|}n |}|}|j|ƒ}||}tj||ƒ}tj|ƒ|jt|ƒt|ƒkr…dS|jdkr¹|jd||d|jƒ|_n"|jd||d|jƒ|_dS(NR$iRC( R RR(R+RRRR<R!R"(R RRRLRMROt z_minus_Mwt denominator((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyR0šs  ,%(RRRR#R0(((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyRzs(Rt __future__RRRtnumpyR(t numpy.linalgRt scipy.linalgRtwarningsRt__all__tobjectRRRR(((sFlib/python2.7/site-packages/scipy/optimize/_hessian_update_strategy.pyts \ˆŠ