ó áp7]c@s~dZddlmZddljZddlZd„Zd„Zdd„Z de fd „ƒYZ e d krzddl jZd ZejjeƒZd gZd ekrÈe eƒeeedƒƒeeedƒƒejejƒejƒƒZeeeƒZejeeƒejeeeƒed ƒejejeƒeje eƒƒdd ƒZejgeD]Z ej!e ƒd ^qtƒZ"ej#ƒejee"ƒeeƒZ$ej#ƒejee$ƒej#ƒej%ed ej&e$ƒej&eƒƒejejƒejƒdƒZ'ej#ƒej%e'd ej&ee'ƒƒej&e'ƒƒejeƒZ(e(dded…Z)ej#ƒej%e)d ej&ee)ƒƒej&e)ƒƒne eƒZ*ee*j+ƒƒee*j,e*j-ƒƒee*j.dddgƒƒee*j,dddddgƒƒejejƒejƒdƒZejejeƒeje eƒƒdd ƒZeeƒZ$ej#ƒejee$ƒeje$eddƒZ/e/e$ƒZ0eje0e$ƒej1eje eƒƒejeƒddddƒZ2e2e$ƒZ3eje3e$ƒedƒedej&e0ƒjƒƒedej&e$ƒjƒƒndS(s from David Huard's scipy sandbox, also attached to a ticket and in the matplotlib-user mailinglist (links ???) Notes ===== out of bounds interpolation raises exception and wouldn't be completely defined :: >>> scoreatpercentile(x, [0,25,50,100]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. >>> percentileofscore(x, [-50, 50]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. idea ==== histogram and empirical interpolated distribution ------------------------------------------------- dual constructor * empirical cdf : cdf on all observations through linear interpolation * binned cdf : based on histogram both should work essentially the same, although pdf of empirical has many spikes, fluctuates a lot - alternative: binning based on interpolated cdf : example in script * ppf: quantileatscore based on interpolated cdf * rvs : generic from ppf * stats, expectation ? how does integration wrt cdf work - theory? Problems * limits, lower and upper bound of support does not work or is undefined with empirical cdf and interpolation * extending bounds ? matlab has pareto tails for empirical distribution, breaks linearity empirical distribution with higher order interpolation ------------------------------------------------------ * should work easily enough with interpolating splines * not piecewise linear * can use pareto (or other) tails * ppf how do I get the inverse function of a higher order spline? Chuck: resample and fit spline to inverse function this will have an approximation error in the inverse function * -> doesn't work: higher order spline doesn't preserve monotonicity see mailing list for response to my question * pmf from derivative available in spline -> forget this and use kernel density estimator instead bootstrap/empirical distribution: --------------------------------- discrete distribution on real line given observations what's defined? * cdf : step function * pmf : points with equal weight 1/nobs * rvs : resampling * ppf : quantileatscore on sample? * moments : from data ? * expectation ? sum_{all observations x} [func(x) * pmf(x)] * similar for discrete distribution on real line * References : ? * what's the point? most of it is trivial, just for the record ? Created on Monday, May 03, 2010, 11:47:03 AM Author: josef-pktd, parts based on David Huard License: BSD iÿÿÿÿ(tprint_functionNcCsMtj|ƒ}t|ƒ}tjtj|ƒtj|ƒƒ}||dƒS(sÄReturn the score at the given percentile of the data. Example: >>> data = randn(100) >>> scoreatpercentile(data, 50) will return the median of sample `data`. gY@(tnptarrayt empiricalcdft interpolatetinterp1dtsort(tdatat percentiletpertcdft interpolator((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pytscoreatpercentileWs  $cCs>t|ƒ}tjtj|ƒtj|ƒƒ}||ƒdS(sDReturn the percentile-position of score relative to data. score: Array of scores at which the percentile is computed. Return percentiles (0-100). Example r = randn(50) x = linspace(-2,2,100) percentileofscore(r,x) Raise an error if the score is outside the range of data. gY@(RRRRR(RtscoreR R ((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pytpercentileofscorees $tHazencCsþtjtj|ƒƒd}t|ƒ}|jƒ}|dkrQ|d|}n©|dkrn||d}nŒ|dkr‹|d|}no|dkr¬|d|d}nN|d krÍ|d|d }n-|d krî|d |d }n tdƒ‚|S(sReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. gð?thazengà?tweibullt californiat chegodayevg333333Ó?gš™™™™™Ù?tcunnanegš™™™™™É?t gringorteng)\Âõ(Ü?g¸…ëQ¸¾?s[Unknown method. Choose among Weibull, Hazen,Chegodayev, Cunnane, Gringorten and California.(Rtargsorttlentlowert ValueError(RtmethodtitNR ((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pyRws"         tHistDistcBsDeZdZd„Zddd„Zd„Zd„Zdd„ZRS( s»Distribution with piecewise linear cdf, pdf is step function can be created from empiricial distribution or from a histogram (not done yet) work in progress, not finished cCsÂtj|ƒ|_tj|jjƒ|jjƒgƒ|_tj|ƒ}|||_tj|ƒ|_ |j ƒ}tj |ƒ|_ t j|j|j ƒ|_t j|j |jƒ|_dS(N(Rt atleast_1dRRtmintmaxtbinlimitRt _datasortedtrankingRRt _empcdfsortedRRtcdfintptppfintp(tselfRtsortindR ((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pyt__init__¥s*  RcCs|dkr!|j}|j}ntjtj|ƒƒd}t|ƒ}|jƒ}|dkrr|d|}n©|dkr||d}nŒ|dkr¬|d|}no|dkrÍ|d|d}nN|d krî|d|d }n-|d kr|d |d }n tdƒ‚|S(sAReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. gð?Rgà?RRRg333333Ó?gš™™™™™Ù?Rgš™™™™™É?Rg)\Âõ(Ü?g¸…ëQ¸¾?s[Unknown method. Choose among Weibull, Hazen,Chegodayev, Cunnane, Gringorten and California.N(tNoneRR#RRRRR(R'RRRRR ((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pyR±s(            cCs |j|ƒS(s& this is score in dh (R%(R'R ((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pytcdf_empÚscCs |j|ƒS(s& this is score in dh (R&(R'tquantile((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pytppf_empâstFreedmancCs¡t|jƒ}|dkrP|jdƒ|jdƒ}d||dd}n1|dkrdtj|jƒ|dd}ntj|jƒ||_|jS( s”Find the optimal number of bins and update the bin countaccordingly. Available methods : Freedman Scott R.gè?gÐ?igð¿itScottgìQ¸…ë @(RRR-RtstdtptpR!tnbin(R'RtnobstIQRtwidth((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pytoptimize_binningìs  %N( t__name__t __module__t__doc__R)R*RR+R-R6(((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pyR›s  )  t__main__idiigà?i2tkigÐ?gè?gà¿gпiiôitsg¸…ëQ¸ž?snegative densitys(np.diff(ppfs)).min()s(np.diff(cdf_ongrid)).min()(4R9t __future__Rtscipy.interpolateRtnumpyRR RRtobjectRR7tmatplotlib.pyplottpyplottpltR3trandomtrandntxtexamplestprinttlinspaceRR txsupptpostplottInterpolatedUnivariateSplineRtempRtxit derivativestpdfemptfiguret cdf_ongridtsteptdifftxsupp2txsotxsthistdR6R+R!R-R&tppfstUnivariateSplinetppfemptppfe(((sElib/python2.7/site-packages/statsmodels/sandbox/stats/stats_dhuard.pytRsn    $d    0/    *! 0 3 "!0   6