B &]\yg @sdZddlmZmZmZddlmZddlmZddl Z ddl m Z ddd d d d gZ Gd ddeZd,ddZd-dd ZeZd.dd Zd/dd ZddZddZddZd0d d Zd1d!d"Zd2d#d$Zd3d'd(Zd4d*d+ZdS)5z Functions --------- .. autosummary:: :toctree: generated/ line_search_armijo line_search_wolfe1 line_search_wolfe2 scalar_search_wolfe1 scalar_search_wolfe2 )divisionprint_functionabsolute_import)warn)minpack2N)xrangeLineSearchWarningline_search_wolfe1line_search_wolfe2scalar_search_wolfe1scalar_search_wolfe2line_search_armijoc@s eZdZdS)rN)__name__ __module__ __qualname__rr8lib/python3.7/site-packages/scipy/optimize/linesearch.pyrsr-C6??2:0yE>+=c  s|dkr }ttr<d} d| fdnd|gdgdg fdd} fdd }t|}t||||||| | | | d \}}}|dd||dfS) a As `scalar_search_wolfe1` but do a line search to direction `pk` Parameters ---------- f : callable Function `f(x)` fprime : callable Gradient of `f` xk : array_like Current point pk : array_like Search direction gfk : array_like, optional Gradient of `f` at point `xk` old_fval : float, optional Value of `f` at point `xk` old_old_fval : float, optional Value of `f` at point preceding `xk` The rest of the parameters are the same as for `scalar_search_wolfe1`. Returns ------- stp, f_count, g_count, fval, old_fval As in `line_search_wolfe1` gval : array Gradient of `f` at the final point NrFTcs&dd7<|fS)Nrrr)s)argsffcpkxkrrphiUszline_search_wolfe1..phicsX|fd<r0dd7<ndtd7<tdS)Nrr)lennpdot)r)rfprimegcgradientgvalnewargsrrrrderphiYs z"line_search_wolfe1..derphi)c1c2amaxaminxtol) isinstancetupler!r"r )rr#rrgfkold_fval old_old_fvalrr)r*r+r,r-epsrr(derphi0stpZfvalr) rrrr#r$r%r&r'rrrr "s&#    c Cs|dkr|d}|dkr |d}|dk rT|dkrTtdd|||} | dkrXd} nd} |} |} tdtj} tdt}d}d }xbt|D]R}t| | | ||| |||| | \}} } }|dd d kr|} ||} ||} qPqWd}|dd d ks |dddkrd}|| |fS)a Scalar function search for alpha that satisfies strong Wolfe conditions alpha > 0 is assumed to be a descent direction. Parameters ---------- phi : callable phi(alpha) Function at point `alpha` derphi : callable dphi(alpha) Derivative `d phi(alpha)/ds`. Returns a scalar. phi0 : float, optional Value of `f` at 0 old_phi0 : float, optional Value of `f` at the previous point derphi0 : float, optional Value `derphi` at 0 c1, c2 : float, optional Wolfe parameters amax, amin : float, optional Maximum and minimum step size xtol : float, optional Relative tolerance for an acceptable step. Returns ------- alpha : float Step size, or None if no suitable step was found phi : float Value of `phi` at the new point `alpha` phi0 : float Value of `phi` at `alpha=0` Notes ----- Uses routine DCSRCH from MINPACK. Ngrg?g)\(@)) sSTARTdr6sFGsERRORsWARN)minr!ZzerosZintcfloatrrZdcsrch)rr(phi0old_phi0r4r)r*r+r,r-alpha1phi1Zderphi1ZisaveZdsaveZtaskmaxiterir5rrrr js8+   $ c  s dgdgdgdg fdd} t trR fddn  fdd|dkr f}t| }dk r fdd }nd}t| ||||| | || d \}}}}|dkrtd tnd}|dd|||fS) a Find alpha that satisfies strong Wolfe conditions. Parameters ---------- f : callable f(x,*args) Objective function. myfprime : callable f'(x,*args) Objective function gradient. xk : ndarray Starting point. pk : ndarray Search direction. gfk : ndarray, optional Gradient value for x=xk (xk being the current parameter estimate). Will be recomputed if omitted. old_fval : float, optional Function value for x=xk. Will be recomputed if omitted. old_old_fval : float, optional Function value for the point preceding x=xk args : tuple, optional Additional arguments passed to objective function. c1 : float, optional Parameter for Armijo condition rule. c2 : float, optional Parameter for curvature condition rule. amax : float, optional Maximum step size extra_condition : callable, optional A callable of the form ``extra_condition(alpha, x, f, g)`` returning a boolean. Arguments are the proposed step ``alpha`` and the corresponding ``x``, ``f`` and ``g`` values. The line search accepts the value of ``alpha`` only if this callable returns ``True``. If the callable returns ``False`` for the step length, the algorithm will continue with new iterates. The callable is only called for iterates satisfying the strong Wolfe conditions. maxiter : int, optional Maximum number of iterations to perform Returns ------- alpha : float or None Alpha for which ``x_new = x0 + alpha * pk``, or None if the line search algorithm did not converge. fc : int Number of function evaluations made. gc : int Number of gradient evaluations made. new_fval : float or None New function value ``f(x_new)=f(x0+alpha*pk)``, or None if the line search algorithm did not converge. old_fval : float Old function value ``f(x0)``. new_slope : float or None The local slope along the search direction at the new value ````, or None if the line search algorithm did not converge. Notes ----- Uses the line search algorithm to enforce strong Wolfe conditions. See Wright and Nocedal, 'Numerical Optimization', 1999, pg. 59-60. For the zoom phase it uses an algorithm by [...]. rNcs&dd7<|fS)Nrrr)alpha)rrrrrrrrszline_search_wolfe2..phicsfdtd7<d}d}|f}||f|d<|d<tdS)Nrr)r r!r")rDr3r#r')rrrr& gval_alphamyfprimerrrrr(s z"line_search_wolfe2..derphicsBdd7<|fd<|d<tdS)Nrr)r!r")rD)rr#r$r&rErrrrr(scs2d|kr||}|||dS)Nrr)rDrx)r(extra_conditionr&rErrrrextra_condition2,s  z,line_search_wolfe2..extra_condition2)rAz*The line search algorithm did not converge)r.r/r!r"r rr)rrFrrr0r1r2rr)r*r+rHrArr4rI alpha_starphi_star derphi_starr) rr(rHrrr#r$r&rErFrrrr s,G    c Cs|dkr|d}|dkr(|dk r(|d}d} |dk rT|dkrTtdd|||} nd} | dkrdd} || } |} |}|dkrdd}x`t| D]<}| dks|dk r| |krd}|}|}d}| dkrd}n d d |}t|tP| ||| |ks| | kr4|d kr4t| | | | |||||||| \}}}P|| }t|| |krj|| | rj| }| }|}P|dkrt| | | | |||||||| \}}}Pd | }|dk rt||}| } |} | } || } |}qW| }| }d}td t||||fS)aFind alpha that satisfies strong Wolfe conditions. alpha > 0 is assumed to be a descent direction. Parameters ---------- phi : callable f(x) Objective scalar function. derphi : callable f'(x), optional Objective function derivative (can be None) phi0 : float, optional Value of phi at s=0 old_phi0 : float, optional Value of phi at previous point derphi0 : float, optional Value of derphi at s=0 c1 : float, optional Parameter for Armijo condition rule. c2 : float, optional Parameter for curvature condition rule. amax : float, optional Maximum step size extra_condition : callable, optional A callable of the form ``extra_condition(alpha, phi_value)`` returning a boolean. The line search accepts the value of ``alpha`` only if this callable returns ``True``. If the callable returns ``False`` for the step length, the algorithm will continue with new iterates. The callable is only called for iterates satisfying the strong Wolfe conditions. maxiter : int, optional Maximum number of iterations to perform Returns ------- alpha_star : float or None Best alpha, or None if the line search algorithm did not converge. phi_star : float phi at alpha_star phi0 : float phi at 0 derphi_star : float or None derphi at alpha_star, or None if the line search algorithm did not converge. Notes ----- Uses the line search algorithm to enforce strong Wolfe conditions. See Wright and Nocedal, 'Numerical Optimization', 1999, pg. 59-60. For the zoom phase it uses an algorithm by [...]. Ngrg?g)\(@cSsdS)NTr)rDrrrrsz&scalar_search_wolfe2..z7Rounding errors prevent the line search from convergingz4The line search algorithm could not find a solution zless than or equal to amax: %srr6z*The line search algorithm did not converge)r;rrr_zoomabs)rr(r=r>r4r)r*r+rHrAalpha0r?phi_a1phi_a0Z derphi_a0rBrJrKrLmsgZ derphi_a1alpha2rrrr Dst;       c Cs*tjddddy|}||}||} || d|| } td} | d| d<|d | d<| d | d<|d| d <t| t|||||||| g\} } | | } | | } | | d| |}|| t|d| }Wntk r d SXWd QRXt|s&d S|S) z Finds the minimizer for a cubic polynomial that goes through the points (a,fa), (b,fb), and (c,fc) with derivative at a of fpa. If no minimizer can be found return None raise)divideoverinvalidr6)r6r6)rr)rr)rr)rrN) r!errstateemptyr"ZasarrayZflattensqrtArithmeticErrorisfinite)afafpabfbcrCdbZdcZdenomZd1ABZradicalxminrrr _cubicmins,      rjc Cstjdddd\y@|}|}||d}||||||}||d|} Wntk rfdSXWdQRXt| sdS| S)z Finds the minimizer for a quadratic polynomial that goes through the points (a,fa), (b,fb) with derivative at a of fpa, rU)rVrWrXg?g@N)r!rZr]r^) r_r`rarbrcDrerfrhrirrr_quadmins  rlc Csd} d} d}d}|}d}x~||}|dkr8||}}n ||}}| dkrf||}t|||||||}| dks|dks|||ks|||kr||}t|||||}|dks|||ks|||kr|d|}||}||| ||ks||kr |}|}|}|}nn||}t|| |kr@| ||r@|}|}|}P|||dkrd|}|}|}|}n|}|}|}|}|}| d7} | | krd}d}d}PqW|||fS)zG Part of the optimization algorithm in `scalar_search_wolfe2`. rCrg?g?Ng?r)rjrlrO)Za_loZa_hiZphi_loZphi_hiZ derphi_lorr(r=r4r)r*rHrArBZdelta1Zdelta2Zphi_recZa_recZdalphar_rbZcchkZa_jZqchkZphi_ajZ derphi_ajZa_starZval_starZ valprime_starrrrrN sb   (   rNrc sjtdgfdd}|dkr6|d} n|} t|} t|| | ||d\} } | d| fS)aMinimize over alpha, the function ``f(xk+alpha pk)``. Parameters ---------- f : callable Function to be minimized. xk : array_like Current point. pk : array_like Search direction. gfk : array_like Gradient of `f` at point `xk`. old_fval : float Value of `f` at point `xk`. args : tuple, optional Optional arguments. c1 : float, optional Value to control stopping criterion. alpha0 : scalar, optional Value of `alpha` at start of the optimization. Returns ------- alpha f_count f_val_at_alpha Notes ----- Uses the interpolation algorithm (Armijo backtracking) as suggested by Wright and Nocedal in 'Numerical Optimization', 1999, pg. 56-57 rcs&dd7<|fS)Nrrr)r?)rrrrrrrrszline_search_armijo..phiNg)r)rP)r!Z atleast_1dr"scalar_search_armijo) rrrr0r1rr)rPrr=r4rDr@r)rrrrrrr ^s"     c Cs0t||||||||d}|d|dd|dfS)z8 Compatibility wrapper for `line_search_armijo` )rr)rPrrr6)r ) rrrr0r1rr)rPrrrrline_search_BFGSsrocCs||}|||||kr$||fS| |dd||||}||}|||||krj||fSx&||kr|d|d||} |d|||||d||||} | | } |d |||||d||||} | | } | tt| dd| |d| } || } | ||| |krT| | fS|| |dksxd| |dkr|d} |}| }|}| }qnWd|fS)a(Minimize over alpha, the function ``phi(alpha)``. Uses the interpolation algorithm (Armijo backtracking) as suggested by Wright and Nocedal in 'Numerical Optimization', 1999, pg. 56-57 alpha > 0 is assumed to be a descent direction. Returns ------- alpha phi1 r6g@rYg@rgQ?N)r!r\rO)rr=r4r)rPr,rRr?rQZfactorr_rbrTZphi_a2rrrrms4",$rm皙??cCs|d}t|} d} d} d} x|| |} || \}}|| ||| d|krX| } P| d||d| d|}|| |} || \}}|| ||| d|kr| } P| d||d| d|}t||| || } t||| || } qW| | ||fS)a> Nonmonotone backtracking line search as described in [1]_ Parameters ---------- f : callable Function returning a tuple ``(f, F)`` where ``f`` is the value of a merit function and ``F`` the residual. x_k : ndarray Initial position d : ndarray Search direction prev_fs : float List of previous merit function values. Should have ``len(prev_fs) <= M`` where ``M`` is the nonmonotonicity window parameter. eta : float Allowed merit function increase, see [1]_ gamma, tau_min, tau_max : float, optional Search parameters, see [1]_ Returns ------- alpha : float Step length xp : ndarray Next position fp : float Merit function value at next position Fp : ndarray Residual at next position References ---------- [1] "Spectral residual method without gradient information for solving large-scale nonlinear systems of equations." W. La Cruz, J.M. Martinez, M. Raydan. Math. Comp. **75**, 1429 (2006). rr6)maxr!clip)rx_kdZprev_fsetagammatau_mintau_maxf_kZf_baralpha_palpha_mrDxpfpFpalpha_tpalpha_tmrrr_nonmonotone_line_search_cruzs*(      r333333?c Cs,d} d} d} x|| |}||\}}||||| d|krH| } P| d||d| d|}|| |}||\}}||||| d|kr| } P| d||d| d|}t||| | | } t||| | | } qW| |d}| |||||}|}| |||||fS)a Nonmonotone line search from [1] Parameters ---------- f : callable Function returning a tuple ``(f, F)`` where ``f`` is the value of a merit function and ``F`` the residual. x_k : ndarray Initial position d : ndarray Search direction f_k : float Initial merit function value C, Q : float Control parameters. On the first iteration, give values Q=1.0, C=f_k eta : float Allowed merit function increase, see [1]_ nu, gamma, tau_min, tau_max : float, optional Search parameters, see [1]_ Returns ------- alpha : float Step length xp : ndarray Next position fp : float Merit function value at next position Fp : ndarray Residual at next position C : float New value for the control parameter C Q : float New value for the control parameter Q References ---------- .. [1] W. Cheng & D.-H. Li, ''A derivative-free nonmonotone line search and its application to the spectral residual method'', IMA J. Numer. Anal. 29, 814 (2009). rr6)r!rt)rrurvr{reQrwrxryrzZnur|r}rDr~rrrrZQ_nextrrr_nonmonotone_line_search_cheng!s,/       r) NNNrrrrrr)NNNrrrrr) NNNrrrNNrC) NNNNrrNNrC)rrr)rrr)rrr)rrprq)rrprqr)__doc__Z __future__rrrwarningsrZscipy.optimizerZnumpyr!Zscipy._lib.sixr__all__RuntimeWarningrr r Z line_searchr r rjrlrNr rormrrrrrr sD    E P   "T 4 ? H