B Yw@sjdZddlmZddlmZddlmZddlmZddlZ ddl Z ddl m Z ddlmZdd lmZdd lmZmZmZmZmZdd lmZd d lmZmZd dlmZmZd dl m!Z!m"Z"m#Z#dgZ$e%e&e j'e j(gZ)e*e j+gZ,ddgZ-dgdgddggZ.GdddeZ/Gddde/Z0Gddde/Z1GdddeZ2Gddde3Z4ddZ5Gdd d e3Z6dS)!zS The classes and functionality used to transform data inputs to consistent types. )absolute_import)copy)chain) itemgetterN) iteritems)zip)HasProps)bokeh_integer_typesDatetimeFloatListString)ColumnDataSource)Column ColumnLabel)BinsStat)collect_attribute_columnsgen_column_namesspecial_columnsZ _charts_onesxycsDeZdZdZeeddZeeddZd fdd Zd dd Z Z S) ColumnAssigneraDefines behavior for assigning columns to dimensions. This class is used to collect assignments between columns and :class:`Builder` dimensions when none are provided. The :class:`ChartDataSource` receives a ColumnAssigner from each :class:`Builder`, which can implement custom behavior. Each subclass must implement the :meth:`get_assignment` method, which returns a `dict` mapping between each dimension in `dims` and one or more column names, or `None` if no assignment is made for the associated dimension. z The list of dimension names that are associated with the :class:`Builder`. The ColumnAssigner should return a dict with each dimension as a key when the :meth:`get_assignment` method is called. )helpz This list of attribute names that are associated with the :class:`Builder`. These can be used to alter which dimensions are assigned which columns, versus which attributes are assigned which columns. Nc s$|dk r||_tt|jf|dS)zCreate the assigner. Args: df (:class:`pandas.DataFrame`, optional): the data source to use for assigning columns from **properties: any attribute of the ColumnAssigner N)_dfsuperr__init__)selfdf properties) __class__3lib/python3.7/site-packages/bkcharts/data_source.pyr6s zColumnAssigner.__init__cCs tddS)Nz3You must return map between each dim and selection.)NotImplementedError)r selectionsr"r"r#get_assignmentCszColumnAssigner.get_assignment)N)N) __name__ __module__ __qualname____doc__r r dimsattrsrr& __classcell__r"r")r!r#rs  rc@seZdZdZdddZdS)OrderedAssignerzAssigns one column for each dimension that is not an attribute, in order. This is the default column assigner for the :class:`Builder`. NcsT|dkstt|dkrLfddjD}ddt|jjDS|SdS)zEGet a mapping between dimension and selection when none are provided.Nrcsg|]}|jkr|qSr")r,).0dim)rr"r# Qsz2OrderedAssigner.get_assignment..cSsi|]\}}||qSr"r")r/r0selr"r"r# Rsz2OrderedAssigner.get_assignment..)lenlistkeysr+rrcolumnstolist)rr%r+r")rr#r&Ns zOrderedAssigner.get_assignment)N)r'r(r)r*r&r"r"r"r#r.Gsr.c@seZdZdZdddZdS)NumericalColumnsAssignerz1Assigns all numerical columns to the y dimension.Ncst|tr |d|d}n dd}i}|j}|j}dk rd|dkrdfdd|D}ndkr|d|dkr||}|d<||d<|S)Nrrcsg|]}|tkr|qSr")r5)r/col)rr"r#r1isz;NumericalColumnsAssigner.get_assignment..index) isinstancedictgetrZ_get_numeric_datar7r8)rr%rrZnum_colsr")rr#r&[s"     z'NumericalColumnsAssigner.get_assignment)N)r'r(r)r*r&r"r"r"r#r9Xsr9c@s0eZdZdZeedddZddZddZdS) DataOperatorzAAn operation that transforms data before it is used for plotting.Nz1 List of columns to perform operation on.)defaultrcCs tddS)Nz3Each data operator must implement the apply method.)r$)rdatar"r"r#applyzszDataOperator.applycCsd|j}d|jj|fS)Nz, z%s(%s))joinr7r!r')rZcol_strr"r"r#__repr__}s zDataOperator.__repr__) r'r(r)r*r rr7rBrDr"r"r"r#r?us r?c@sXeZdZdZddZddZeddZdd Zd d Z d d Z eddZ ddZ dS) DataGroupzContains subset of data and metadata about it. The DataGroup contains a map from the labels of each attribute associated with an :class:`AttrSpec` to the value of the attribute assigned to the DataGroup. cCs||_||_||_dS)aCreate a DataGroup for the data, with a label and associated attributes. Args: label (str): the label for the group based on unique values of each column data (:class:`pandas.DataFrame`): the subset of data associated with the group attr_specs dict(str, :class:`AttrSpec`): mapping between attribute name and the associated :class:`AttrSpec`. N)labelrA attr_specs)rrFrArGr"r"r#rs zDataGroup.__init__cCs`t|tr|j|St|tr8t|dkr8|j|dSt|trXt|dkrX|j|SdSdS)zGet the data associated with the selection of columns. Args: selection (List(Str) or Str): the column or columns selected Returns: :class:`pandas.DataFrame` rrN)r<strrAr5r4)r selectionr"r"r# get_valuess   zDataGroup.get_valuescCs t|jS)z>The :class:`ColumnDataSource` representation of the DataFrame.)rrA)rr"r"r#sourceszDataGroup.sourcecCs |j|S)zCGet the value of the :class:`AttrSpec` associated with `spec_name`.)rG)r spec_namer"r"r# __getitem__szDataGroup.__getitem__cCsdt|j|jfS)Nz )rHrFrG)rr"r"r#rDszDataGroup.__repr__cCs t|jjS)N)r4rAr;)rr"r"r#__len__szDataGroup.__len__cCst|jS)N)r5rGr6)rr"r"r# attributesszDataGroup.attributescCsFi}|jdk r.||jt|j|d<nd|d<||j|S)NZ chart_index)rFupdatetupleitemsrG)rrowr"r"r#to_dicts   zDataGroup.to_dictN) r'r(r)r*rrJpropertyrKrMrDrNrOrTr"r"r"r#rEs  rEc +sFtf|r x|jddD]\}}i}i}xt|D]\}}|jdk rtfdd|jD}t|trt||} t|} n |} d} t| ts| f} t| tst| ts| g} x$t| | D]\} } | || <qWnd} || ||<q:Wt |||dVq Wn8i}x"t|D]\}}|d||<qWt d||dVdS)aConvenience iterator around pandas groupby and attribute specs. Args: df (:class:`~pandas.DataFrame`): The entire data source being used for the Chart. **specs: Name, :class:`AttrSpec` pairing, used to identify the lowest level where the data is grouped. Yields: :class:`DataGroup`: each unique group of data to be used to produce glyphs F)sortNcsg|]}|qSr")r;)r/r:) spec_colsr"r#r1szgroupby..r)rFrArG) rgroupbyrr7rQr<rr5rrE) rspecsnamerAr,Z group_labelrLspecZname_idxrFcolsr:valuer")rWr#rXs4     rXc@sTeZdZdZdddedfddZeddZddZd d Z d d Z d dZ ddZ d?ddZ ddZddZeddZeddZeddZedd Zed!d"Zed@d#d$Zed%d&Zed'd(Zed)d*Zed+d,Zed-d.Zd/d0Zed1d2Zed3d4Zed5d6Z ed7d8Z!ed9d:Z"ed;d<Z#ed=d>Z$dS)AChartDataSourceaValidates, normalizes, groups, and assigns Chart attributes to groups. Supported inputs are: - **Array-like**: list, tuple, :class:`numpy.ndarray`, :class:`pandas.Series` - **Table-like**: - records: list(dict) - columns: dict(list), :class:`pandas.DataFrame`, or blaze resource Converts inputs that could be treated as table-like data to pandas DataFrame, which is used for assigning attributes to data groups. NcKs|dkr t}|dkrt}|dd|_|p,g|_|jdd|_||_g|_||_ ||jt |j|jd|_ |j |f||_ |||||_|dS)a5Create a :class:`ChartDataSource`. Args: df (:class:`pandas.DataFrame`): the original data source for the chart dims (List(Str), optional): list of valid dimensions for the chart. required_dims (List(List(Str)), optional): list of list of valid dimensional selections for the chart. selections (Dict(String, List(Column)), optional): mapping between a dimension and the column name(s) associated with it. This represents what the user selected for the current chart. column_assigner (:class:`ColumnAssigner`, optional): a reference to a ColumnAssigner class, which is used to collect dimension column assignment when keyword arguments aren't provided. The default value is :class:`OrderedAssigner`, which assumes you want to assign each column or array to each dimension of the chart in order that they are received. attrs (list(str)): list of attribute names the chart uses N input_typeF)Zdeep)rr+r,) DEFAULT_DIMSDEFAULT_REQ_DIMSpopr_r,r_data_dims operations_required_dimsr5column_assignerget_selections _selectionssetup_derived_columnsapply_operationscollect_metadatameta_validate_selections)rrr+ required_dimsr%rgr,kwargsr"r"r#rs&   zChartDataSource.__init__csfddtjDS)Ncs i|]\}}|jkr||qSr")r,)r/r0val)rr"r#r3Isz.ChartDataSource.attr_specs..)rri)rr")rr#rGGszChartDataSource.attr_specscKsi}x*|jD] }||d}|dk r |||<q Wt|dkr~|dkrT|j}qt|trtt|dkr||}qtdn|jj|d}x|jD]}||krd||<qW|S)a Maps chart dimensions to selections and checks input requirements. Returns: mapping between each dimension and the selected columns. If no selection is made for a dimension, then the dimension will be associated with `None`. Nrzfselections input must be provided as: dict(dimension: column) or None)r%) rdrbr4r6rgr&r<r= ValueError)rr%rpZ select_mapr0Z dim_selectr"r"r#rhKs$       zChartDataSource.get_selectionscCs|j}xt|jD]r\}}t|tr>|||_|j||<t|tr|t|t rj|||_|j ||<nt d|j |f|j |qW||_dS)zApplies each data operation.z)Stat input of %s for %s is not supported.N)rirrr<r?rBrcrZrrZcenters_column TypeErrorr!reappend)rr%r0Zselectr"r"r#rkus        z ChartDataSource.apply_operationscCsZxT|jD]J}||}|dk rt|tr|tkr||jjkrt||j|j|<qWdS)zEAttempt to add special case columns to the DataFrame for the builder.N)rdr<rHrrr7r8rc)rr0Z dim_selectionr"r"r#rjs  z%ChartDataSource.setup_derived_columnscCs||jkr|j|SdSdS)zGet the columns selected for the given dimension name. e.g. dim='x' Returns: the columns selected as a str or list(str). If the dimension is not in `_selections`, `None` is returned. N)ri)rr0r"r"r#rMs  zChartDataSource.__getitem__cCs||j|<|dS)N)rirj)rr0r]r"r"r# __setitem__s zChartDataSource.__setitem__variabler]csxjD]}j|}t|tr(|j}n|}|jjkrD|d7}||kr|j|<|dk rtfdd|Drtj j ||||d_ qt t j jt |}tj j ||||d_ qWdS)a De-pivots `_data` from a 'wide' to 'tall' layout. A wide table is one where the column names represent a categorical variable and each contains only the values associated with each unique value of the categorical variable. This method uses the :func:`pandas.melt` function with additional logic to make sure that the same data source can have multiple operations applied, and so all other columns are maintained through the stacking process. Example: .. note:: This example is fairly low level and is not something the typical user should worry about. The interface for data transformations from the user perspective are the :ref:`bkcharts_functions`. >>> data = {'a': [1, 2, 3, 4], ... 'b': [2, 3, 4, 5], ... 'month': ['jan', 'jan', 'feb', 'feb'] ... } >>> ds = ChartDataSource.from_data(data) >>> ds['x'] =['a', 'b'] # say we selected a and b for dimension x We may want to combine 'a' and 'b' together. The final data would look like the following: >>> ds.stack_measures(['c', 'd'], var_name='c_d_variable', ... value_name='c_d_value') >>> ds.df Out[35]: month a_b_variable a_b_value 0 jan a 1 1 jan a 2 2 feb a 3 3 feb a 4 4 jan b 2 5 jan b 3 6 feb b 4 7 feb b 5 The transformed data will use the `var_name` and `value_name` inputs to name the columns. These derived columns can then be used as a single column to reference the values and the labels of the data. In the example, I could plot a_b_value vs month, and color by a_b_variable. What this does for you over the :meth:`pandas.melt` method is that it will apply the :class:`DataOperator` for a dimension if it exists (e.g. :class:`Blend`, generated by :func:`blend`), and it will try to handle the id columns for you so you don't lose other columns with the melt transformation. Returns: None _Ncsg|]}|jjkqSr")rr7r8)r/Zmeasure)rr"r#r1sz2ChartDataSource.stack_measures..)Zid_varsZ value_varsvar_name value_name) rdrir<r?r7rr8allpdZmeltrcr5set)rZmeasuresZidsrxryr0rIdim_colsr")rr#stack_measuress <      zChartDataSource.stack_measurescKs"t|dkrtdt|jf|S)aN Iterable of chart attribute specifications, associated with columns. Iterates over DataGroup, which represent the lowest level of data that is assigned to the attributes for plotting. Yields: a DataGroup, which contains metadata and attributes assigned to the group of data rzBYou must provide one or more Attribute Specs to support iteration.)r4rrrXrc)rrYr"r"r#rXs zChartDataSource.groupbyc Ks|j}|s|j}g}g}d}x:|jf|D]*}|jdkr@d}||||q.W|rtj |dg}d|d<d|d<tj ||dd}|d=n0tj |}t |dj }tj ||d|d }|S) alProduce new DataFrame from source data and `AttrSpec` provided. Args: **attr_specs (str, `AttrSpec`, optional): pairs of names and attribute spec objects. This is optional and not required only if the `ChartDataSource` already contains references to the attribute specs. Returns: pd.DataFrame: a new dataframe that includes a column for each of the attribute specs joined in, plus one special column called `chart_index`, which contains the unique items between the different attribute specs. FNTrZ join_valueZ join_column)onleft)Zhowr) rcrrGrXrFrtrTr{ DataFrame from_recordsmerger5r6) rrGrgroupsZrowsZno_indexgroupZ attr_datar\r"r"r# join_attrss*    zChartDataSource.join_attrsc sdddk r&fdd|D}fdd|D}fdd|D}t|dkrjt|dkrjtdt|dkrt|dkrfd dtD}t|dkrfd d|D}|rtt|}tt|} }x|D]}g} |} | s(t fd d| Ds(| g} xP| D]H} x@t ||D]2\} } t j t | t | kr>| | q>Wq.Wt| d kr| d} | ||<qW||d <nfdd|D}dk rΈd<t|dkrdd<j|fSt|dkrt|d krtdn|d}t|trvt fdd|Drldd<fdt jj|diStdnh|rdd<fdt jj|diSt|t jrdd<fd|iStdt|dS)a1Automatically handle all valid inputs. Attempts to use any data that can be represented in a Table-like format, along with any generated requirements, to produce a :class:`ChartDataSource`. Internally, these data types are generated, so that a :class:`pandas.DataFrame` can be generated. Identifies inputs that are array vs table like, handling them accordingly. If possible, existing column names are used, otherwise column names are generated. Returns: :class:`ColumnDataSource` r,Ncs,g|]$}t|tr$t|tr|kr|qSr")r<rH)r/arg)r,r"r#r1Ws z-ChartDataSource.from_data..csg|]}|r|qSr")is_array)r/r)clsr"r#r1Zscs$g|]}|s|r|qSr")is_table is_list_dicts)r/r)rr"r#r1[srz&Only input either array or table data.cs8g|]0\}}|s |r|dk r|dk r|qS)r+ro)is_list_arraysr)r/kv)rr"r#r1es csg|] }|qSr"r")r/r0)rpr"r#r1iscsg|]}|qSr")r)r/ dim_input)rr"r#r1ssrr7csg|]}|r|qSr")r)r/r)rr"r#r1sZ iter_arrayr_zInput a single table data type.csg|]}|qSr")r)r/r:)rr"r#r1sZ dict_arrayr)rAz1Input of table-like dict must be column-oriented.Z list_dictsrz=Unable to recognize inputs for conversion to dataframe for %s)rbr4rsrrr5r from_iterablerrrzrr{Seriesrt from_arraysr<r=valuesr from_dictrrtype)rargsrparraysZtablesZ list_dimsZ col_namesZ new_kwargsr0r}Z dim_inputsrarrayZcol_nametabler")r,rrpr# from_data@sl             zChartDataSource.from_datacCsd}t|trPt|dkrPt|dtrPt|ddtsPt|ddsP|St|trrtdd|Drd}nt|tjrt|jdkrd}|S)z_Verify if input data is a list of array-like data. Returns: bool FrrcSsg|]}t|qSr")r^r)r/r:r"r"r#r1sz2ChartDataSource.is_list_arrays..T) r<r5r4r^rrznpndarrayshape)rAZvalidr"r"r#rs$$  zChartDataSource.is_list_arrayscCs|jS)N)rc)rr"r"r#rszChartDataSource.dfcCs t|jS)N)rr)rr"r"r#rKszChartDataSource.sourcecKs||d}|sdS|SdS)zReturns dimensions by name from kwargs. Returns: iterable(str): iterable of dimension names as strings N)rr)rb)rpr+r"r"r#_collect_dimensionss z#ChartDataSource._collect_dimensionsc  stfdd|Drt|}tt|}|p:tt|}t||dt}xBt ||D]"\}}fdd|D}|||<q\Wn|ptt|}x>t |D]2\} } t | t j r| j} | |kr| dk r| || <qWddt ||D} fd t jj| d i|S) zyProduce :class:`ColumnDataSource` from array-like data. Returns: :class:`ColumnDataSource` c3s|]}|VqdS)N)r)r/r)rr"r# sz.ChartDataSource.from_arrays..r+csg|]}dqS)r)rb)r/rw)r\r"r#r1sz/ChartDataSource.from_arrays..NcSsi|]\}}||qSr"r")r/Z column_namerr"r"r#r3sz/ChartDataSource.from_arrays..r)rA)anyrr5rrrr4r>r`r enumerater<r{rrZrr) rrZ column_namesrpZlist_of_arraysr+r0Z list_of_arrayr2irrZrr")rr\r#rs"    zChartDataSource.from_arrayscKs|fdtj|i|S)zyProduce :class:`ColumnDataSource` from table-like dict. Returns: :class:`ColumnDataSource` r)r{rr)rrArpr"r"r#r szChartDataSource.from_dictcCst|tpt|S)z}Verify if data is table-like. Inspects the types and structure of data. Returns: bool )r^ _is_valid TABLE_TYPESr)rAr"r"r#rs zChartDataSource.is_tablecCst|totdd|DS)z]Verify if data is row-oriented, table-like data. Returns: bool cSsg|]}t|tqSr")r<r=)r/rSr"r"r#r1*sz1ChartDataSource.is_list_dicts..)r<r5rz)rAr"r"r#r"szChartDataSource.is_list_dictscCst|rdSt|tSdS)zJVerify if data is array-like. Returns: bool FN)r^rr ARRAY_TYPES)rAr"r"r#r,s zChartDataSource.is_arraycstfdd|DS)zChecks for each type against data. Args: data: a generic source of data types: a list of classes Returns: bool csg|]}t|qSr")r<)r/Z valid_type)rAr"r#r1Fsz-ChartDataSource._is_valid..)r)rAtypesr")rAr#r:s zChartDataSource._is_validcsj}j}ddt|D}jdk r:fdd|D}t|dkrx8|D]0t|tkrbqLtfdd|DrLdSqLWd}dd|D}d dt|D}t|d |d |d jfndSdS) zyRaises selection error if selections are not valid compared to requirements. Returns: None cSsg|]\}}|dk r|qS)Nr")r/r0r2r"r"r#r1Rsz8ChartDataSource._validate_selections..Ncsg|]}|jkr|qSr")r,)r/r0)rr"r#r1Tsrcsg|] }|kqSr"r")r/r0)reqr"r#r1_szDid not receive a valid combination of selections. Valid configurations are: %s Received inputs are: %s Available columns are: %scSs g|]}ddd|DqS)z and cSsg|] }d|qS)z%s = r")r/r0r"r"r#r1hszCChartDataSource._validate_selections...)rC)r/Z required_dimr"r"r#r1hscSs,g|]$\}}|dk rdt|t|fqS)Nz%s = %s)rH)r/r0r2r"r"r#r1jsz or z, ) rfrirr,r4rzrrrCr7)rror%r+Z error_strZreq_strZ selection_strr")rrr#rnHs&    $z$ChartDataSource._validate_selectionscCs2t|tjrtt|Stft}t||SdS)zUVerifies that value is a numerical type. Returns: bool N)r<r{rrr Zis_validfloatr )r]Znumbersr"r"r# is_numberss  zChartDataSource.is_numbercCs*yt|}|dStk r$dSXdS)zuVerifies that value is a valid Datetime type, or can be converted to it. Returns: bool TFN)r rr)r]Zdtr"r"r# is_datetimes zChartDataSource.is_datetimecCsiS)z6Introspect which columns match to which types of data.r")rAr"r"r#rlsz ChartDataSource.collect_metadatacCs|jjS)z\All column names associated with the data. Returns: List(Str) )rcr7)rr"r"r#r7szChartDataSource.columnscCs|jjS)z8The index for the :class:`pandas.DataFrame` data source.)rcr;)rr"r"r#r;szChartDataSource.indexcCs|jjS)N)rcr)rr"r"r#rszChartDataSource.valuescCs|tkr dSdSdS)zmVerify if the column provided matches to known computed columns. Returns: bool TFN)COMPUTED_COLUMN_NAMES)columnr"r"r# is_computedszChartDataSource.is_computed)Nrvr])N)%r'r(r)r*r.rrUrGrhrkrjrMrur~rXr classmethodr staticmethodrrrKrrrrrrrrnrrrlr7r;rrr"r"r"r#r^ sB * *  X- x      +     r^)7r*Z __future__rr itertoolsroperatorrZnumpyrZpandasr{ZsixrZ six.movesrZbokeh.core.has_propsrZbokeh.core.propertiesr r r r r Zbokeh.models.sourcesrr rrZstatsrrZutilsrrrrrQr5rrrr=rrr`rarr.r9r?objectrErXr^r"r"r"r#s4         ( HC