B Zg= @sXdZddlmZddlmZddlZejZ dZ ddZ ddid fd d Z ddifd d Z ddifddZdde dddedddddde_ddid fddZe edddddde_ddid fddZe edddd d!de_ddifd"d#Ze ed$ddd%d&de_eZejd'7_ed(krTddlmZdd)lmZddlZejjZej ej!d d*e_!e"ej#ej!Z$e$j%d+d,Z&dd-d.d/gZ'e$j(Z)e$j*Z*e$j+Z,dRd2d3Z-d4d5Z.d6d7Z/d8d9Z0d:Z1e2e1d;3e1d<Z4ej56e1d;Z4e7dd=d;gZ8e7d>d>d>gZ8e8Z9e:e4e9d?ej56e1Z;e:ej<=e4e;Z8d1Z>e;e4fZ?dd@l@mAZAeABe0dAe?ZCeDe dBe.e>e4e e8e/e>e?ZEe e8e/e> e?ZFeDeEGdeDdCeDe:eEjHeEeDdDeDd=e:e4jHe4eEeFdEZIeDe:eIjHeIeDeJe8e0dFe?eJe8e0de?ZKeDdGeDeKeDdHdeDeKdd=e:e4jHe4x@dID]8ZeDdJeeDeJe8e0ee?dd=e:e4jHe4qbWee8e0e?dKZLeDdLeDeLd=e:e4jHe4ee8e0e?dKZMeDdMeDeMd=e:e4jHe4ddlNZOeOPdNdOZQeQe8ZReDdPeDeRd=e:e4jHe4eOSdQdOZTeTe8ZUdS)Saxnumerical differentiation function, gradient, Jacobian, and Hessian Author : josef-pkt License : BSD Notes ----- These are simple forward differentiation, so that we have them available without dependencies. * Jacobian should be faster than numdifftools because it doesn't use loop over observations. * numerical precision will vary and depend on the choice of stepsizes )print_function)rangeNa Calculate Hessian with finite difference derivative approximation Parameters ---------- x : array_like value at which function derivative is evaluated f : function function of one array f(x, `*args`, `**kwargs`) epsilon : float or array-like, optional Stepsize used, if None, then stepsize is automatically chosen according to EPS**(1/%(scale)s)*x. args : tuple Arguments for function `f`. kwargs : dict Keyword arguments for function `f`. %(extra_params)s Returns ------- hess : ndarray array of partial second derivatives, Hessian %(extra_returns)s Notes ----- Equation (%(equation_number)s) in Ridout. Computes the Hessian as:: %(equation)s where e[j] is a vector with element j == 1 and the rest are zero and d[i] is epsilon[i]. References ----------: Ridout, M.S. (2009) Statistical applications of the complex-step method of numerical differentiation. The American Statistician, 63, 66-74 cCsj|dkr(td|tt|d}n>t|rHt|}||nt|}|j|jkrft d|S)Ng?g?z6If h is not a scalar it must have the same shape as x.) EPSnpZmaximumabsZisscalaremptyZfillZasarrayshape ValueError)xsepsilonnhr8lib/python3.7/site-packages/statsmodels/tools/numdiff.py _get_epsilon^s      rrFc Cs:t|}||f||}t|j}t|f|tt|j} t|ft} |st|d||}xt |D]D} || | | <||| f||||| | | ddf<d| | <qjWn|t|d||d}xht t|D]X} || | | <||| f||||| f||d|| | | ddf<d| | <qW| j S)aN Gradient of function, or Jacobian if function f returns 1d array Parameters ---------- x : array parameters at which the derivative is evaluated f : function `f(*((x,)+args), **kwargs)` returning either one value or 1d array epsilon : float, optional Stepsize, if None, optimal stepsize is used. This is EPS**(1/2)*x for `centered` == False and EPS**(1/3)*x for `centered` == True. args : tuple Tuple of additional arguments for function `f`. kwargs : dict Dictionary of additional keyword arguments for function `f`. centered : bool Whether central difference should be returned. If not, does forward differencing. Returns ------- grad : array gradient or Jacobian Notes ----- If f returns a 1d array, it returns a Jacobian. If a 2d array is returned by f (e.g., with a value for each observation), it returns a 3d array with the Jacobian of each observation with shape xk x nobs x xk. I.e., the Jacobian of the first observation would be [:, 0, :] Ngg@) lenrZ atleast_1drzerosZ promote_typesfloatZdtyperrZsqueezeT) r fr argskwargsZcenteredr f0ZdimgradZeikrrr approx_fprimems$!  , . rcsRt}td|t|d}fddt|D}t|jS)a Calculate gradient or Jacobian with complex step derivative approximation Parameters ---------- x : array parameters at which the derivative is evaluated f : function `f(*((x,)+args), **kwargs)` returning either one value or 1d array epsilon : float, optional Stepsize, if None, optimal stepsize is used. Optimal step-size is EPS*x. See note. args : tuple Tuple of additional arguments for function `f`. kwargs : dict Dictionary of additional keyword arguments for function `f`. Returns ------- partials : ndarray array of partial derivatives, Gradient or Jacobian Notes ----- The complex-step derivative has truncation error O(epsilon**2), so truncation error can be eliminated by choosing epsilon to be very small. The complex-step derivative avoids the problem of round-off error with small epsilon because there is no subtraction. y?cs.g|]&\}}|fj|qSr)imag).0iZih)rr rrr rr sz$approx_fprime_cs..)rrrZidentity enumeratearrayr)r rr rrr Z incrementsZpartialsr)rr rrr rapprox_fprime_css ! r&c Cst|}t|d||}t|}t||}t|}xt|D]} xt| |D]} ||d|| ddf|| ddff||||d|| ddf|| ddff||jd|| | f|| | f<|| | f|| | f<qNWq>W|S)aCalculate Hessian with complex-step derivative approximation Parameters ---------- x : array_like value at which function derivative is evaluated f : function function of one array f(x) epsilon : float stepsize, if None, then stepsize is automatically chosen Returns ------- hess : ndarray array of partial second derivatives, Hessian Notes ----- based on equation 10 in M. S. RIDOUT: Statistical Applications of the Complex-step Method of Numerical Differentiation, University of Kent, Canterbury, Kent, U.K. The stepsize is the same for the complex and the finite difference part. ry?Ng@)rrrdiagouterrr ) r rr rrr reehessr"jrrrapprox_hess_css  r,z=Calculate Hessian with complex-step derivative approximation  r3Z10zk1/(2*d_j*d_k) * imag(f(x + i*d[j]*e[j] + d[k]*e[k]) - f(x + i*d[j]*e[j] - d[k]*e[k])) )ZscaleZ extra_paramsZ extra_returnsZequation_numberZequationcCs0t|}t|d||}t|}||f||} t|} x4t|D](} |||| ddff||| | <qDWt||} xt|D]} x~t| |D]p} |||| ddf|| ddff||| | | | | | | | f| | | f<| | | f| | | f<qWqW|r(| | |}| |fS| SdS)Nr)rrrr'rrr()r rr rr return_gradr rr)rgr"r*r+rrrr approx_hess1s   ( X r2zFreturn_grad : bool Whether or not to also return the gradient z7grad : nparray Gradient if return_grad == True 7zB1/(d_j*d_k) * ((f(x + d[j]*e[j] + d[k]*e[k]) - f(x + d[j]*e[j]))) c Cst|}t|d||}t|}||f||} t|} t|} xXt|D]L} |||| ddff||| | <|||| ddff||| | <qNWt||} xt|D]} xt| |D]}|||| ddf||ddff||| | | || |||| ddf||ddff||| | | || d| | |f| | |f<| | |f| || f<qWqW|r| | |}| |fS| SdS)Nrr)rrrr'rrr()r rr rrr0r rr)rr1Zggr"r*r+rrrr approx_hess2(s$   $(  r48z1/(2*d_j*d_k) * ((f(x + d[j]*e[j] + d[k]*e[k]) - f(x + d[j]*e[j])) - (f(x + d[k]*e[k]) - f(x)) + (f(x - d[j]*e[j] - d[k]*e[k]) - f(x + d[j]*e[j])) - (f(x - d[k]*e[k]) - f(x))) c CsFt|}t|d||}t|}t||}xt|D]} xt| |D]} |||| ddf|| ddff|||||| ddf|| ddff|||||| ddf|| ddff|||||| ddf|| ddff||d|| | f|| | f<|| | f|| | f<qJWq8W|S)Ng@)rrrr'r(r) r rr rrr rr)r*r"r+rrr approx_hess3Us  2r749a 1/(4*d_j*d_k) * ((f(x + d[j]*e[j] + d[k]*e[k]) - f(x + d[j]*e[j] - d[k]*e[k])) - (f(x - d[j]*e[j] + d[k]*e[k]) - f(x - d[j]*e[j] - d[k]*e[k]))z& This is an alias for approx_hess3__main__)approx_fhess_p)ZprependZnewton)methodg?gffffff?i#B ;ư>cs,fdd}t|||t||| dS)Ncst|ddS)Ng#B ;)rr)r&)r )rfuncrrrsz approx_hess_cs_old..gradg@)r)r r?rrr rr)rr?rapprox_hess_cs_olds r@cCst||dS)Nr)rdotsum)betar rrrfunsrDcCst||}||dS)Nr)rrA)rCyr Zxbrrrfun1s rFcCst|||dS)Nr)rFrB)rCrEr rrrfun2srGrrg?g?)optimize)rrr)rrrz np.dot(jac.T, jac)z 2*np.dot(x.T, x)g@gMbP?Zhessfdz epsilon =)gMbP?g-C6?gh㈵>gư>zeps =)rhcs2hfd3cCs t|ttS)N)rGrEr )arrrsrNZnumdiffcCs t|ttS)N)rGrEr )rMrrrrNs)rr=r>)V__doc__Z __future__rZstatsmodels.compat.pythonrZnumpyrZMachArZepsrZ _hessian_docsrrr&r,joinsplitdictr2r4r7Z approx_hess__name__Zstatsmodels.apiZapiZsmZscipy.optimize.optimizer;ZdatasetsZspectorloaddataZ add_constantZexogZProbitZendogmodZfitZresZ test_paramsZloglikeZllfZscoreZhessianr*r@rDrFrGZnobsZarangeZreshaper ZrandomZrandnr%ZxkrCrArEZlinalgZpinvr rZscipyrJZfminZxfminprintZjacZjacminrBrZjac2Zapprox_hess_oldZherKrLZ numdifftoolsZndZHessianZhndZhessndZGradientZgndZgradndrrrrs   (7*)               .