CrossValidator#

class pyspark.ml.connect.tuning.CrossValidator(*, estimator=None, estimatorParamMaps=None, evaluator=None, numFolds=3, seed=None, parallelism=1, foldCol='')[source]#

K-fold cross validation performs model selection by splitting the dataset into a set of non-overlapping randomly partitioned folds which are used as separate training and test datasets e.g., with k=3 folds, K-fold cross validation will generate 3 (training, test) dataset pairs, each of which uses 2/3 of the data for training and 1/3 for testing. Each fold is used as the test set exactly once.

New in version 3.5.0.

Examples

>>> from pyspark.ml.connect.tuning import CrossValidator
>>> from pyspark.ml.connect.classification import LogisticRegression
>>> from pyspark.ml.connect.evaluation import BinaryClassificationEvaluator
>>> from pyspark.ml.tuning import ParamGridBuilder
>>> from sklearn.datasets import load_breast_cancer
>>> lor = LogisticRegression(maxIter=20, learningRate=0.01)
>>> ev = BinaryClassificationEvaluator()
>>> grid = ParamGridBuilder().addGrid(lor.maxIter, [2, 20]).build()
>>> cv = CrossValidator(estimator=lor, evaluator=ev, estimatorParamMaps=grid)
>>> sk_dataset = load_breast_cancer()
>>> train_dataset = spark.createDataFrame(
...     zip(sk_dataset.data.tolist(), [int(t) for t in sk_dataset.target]),
...     schema="features: array<double>, label: long",
... )
>>> cv_model = cv.fit(train_dataset)
>>> transformed_dataset = cv_model.transform(train_dataset.limit(10))
>>> cv_model.avgMetrics
[0.5527792527167658, 0.8348714668615984]
>>> cv_model.stdMetrics
[0.04902833489813031, 0.05247132866444953]

Methods

`clear`(param)	Clears a param from the param map if it has been explicitly set.
`copy`([extra])	Creates a copy of this instance with a randomly generated uid and some extra params.
`explainParam`(param)	Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
`explainParams`()	Returns the documentation of all params with their optionally default values and user-supplied values.
`extractParamMap`([extra])	Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
`fit`(dataset[, params])	Fits a model to the input dataset with optional parameters.
`getEstimator`()	Gets the value of estimator or its default value.
`getEstimatorParamMaps`()	Gets the value of estimatorParamMaps or its default value.
`getEvaluator`()	Gets the value of evaluator or its default value.
`getFoldCol`()	Gets the value of foldCol or its default value.
`getNumFolds`()	Gets the value of numFolds or its default value.
`getOrDefault`(param)	Gets the value of a param in the user-supplied param map or its default value.
`getParallelism`()	Gets the value of parallelism or its default value.
`getParam`(paramName)	Gets a param by its name.
`getSeed`()	Gets the value of seed or its default value.
`get_uid_map`(instance)
`hasDefault`(param)	Checks whether a param has a default value.
`hasParam`(paramName)	Tests whether this instance contains a param with a given (string) name.
`isDefined`(param)	Checks whether a param is explicitly set by user or has a default value.
`isSet`(param)	Checks whether a param is explicitly set by user.
`load`(path)	Load Estimator / Transformer / Model / Evaluator from provided cloud storage path.
`loadFromLocal`(path)	Load Estimator / Transformer / Model / Evaluator from provided local path.
`save`(path, *[, overwrite])	Save Estimator / Transformer / Model / Evaluator to provided cloud storage path.
`saveToLocal`(path, *[, overwrite])	Save Estimator / Transformer / Model / Evaluator to provided local path.
`set`(param, value)	Sets a parameter in the embedded param map.
`setCollectSubModels`(value)	Sets the value of `collectSubModels`.
`setEstimator`(value)	Sets the value of `estimator`.
`setEstimatorParamMaps`(value)	Sets the value of `estimatorParamMaps`.
`setEvaluator`(value)	Sets the value of `evaluator`.
`setFoldCol`(value)	Sets the value of `foldCol`.
`setNumFolds`(value)	Sets the value of `numFolds`.
`setParallelism`(value)	Sets the value of `parallelism`.
`setParams`(*[, estimator, ...])	setParams(self, *, estimator=None, estimatorParamMaps=None, evaluator=None, numFolds=3, seed=None, parallelism=1, collectSubModels=False, foldCol=""): Sets params for cross validator.
`setSeed`(value)	Sets the value of `seed`.

Attributes

`estimator`
`estimatorParamMaps`
`evaluator`
`foldCol`
`numFolds`
`parallelism`
`params`	Returns all params ordered by name.
`seed`

Methods Documentation

clear(param)#: Clears a param from the param map if it has been explicitly set.

copy(extra=None)[source]#

Creates a copy of this instance with a randomly generated uid and some extra params. This copies creates a deep copy of the embedded paramMap, and copies the embedded and extra parameters over.

New in version 3.5.0.

Parameters

extradict, optional: Extra parameters to copy to the new instance

Returns

CrossValidator: Copy of this instance

explainParam(param)#: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams()#: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra=None)#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters

extradict, optional: extra param values

Returns

dict: merged param map

fit(dataset, params=None)#

Fits a model to the input dataset with optional parameters.

New in version 3.5.0.

Parameters

datasetpyspark.sql.DataFrame or py:class:pandas.DataFrame: input dataset, it can be either pandas dataframe or spark dataframe.
paramsa dict of param values, optional: an optional param map that overrides embedded params.

Returns

Transformer: fitted model

getEstimator()#: Gets the value of estimator or its default value.

New in version 2.0.0.

getEstimatorParamMaps()#: Gets the value of estimatorParamMaps or its default value.

New in version 2.0.0.

getEvaluator()#: Gets the value of evaluator or its default value.

New in version 2.0.0.

getFoldCol()#: Gets the value of foldCol or its default value.

New in version 3.1.0.

getNumFolds()#: Gets the value of numFolds or its default value.

New in version 1.4.0.

getOrDefault(param)#: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getParallelism()#: Gets the value of parallelism or its default value.

getParam(paramName)#: Gets a param by its name.

getSeed()#: Gets the value of seed or its default value.

static get_uid_map(instance)#

hasDefault(param)#: Checks whether a param has a default value.

hasParam(paramName)#: Tests whether this instance contains a param with a given (string) name.

isDefined(param)#: Checks whether a param is explicitly set by user or has a default value.

isSet(param)#: Checks whether a param is explicitly set by user.

classmethod load(path)#: Load Estimator / Transformer / Model / Evaluator from provided cloud storage path.

New in version 3.5.0.

classmethod loadFromLocal(path)#: Load Estimator / Transformer / Model / Evaluator from provided local path.

New in version 3.5.0.

save(path, *, overwrite=False)#: Save Estimator / Transformer / Model / Evaluator to provided cloud storage path.

New in version 3.5.0.

saveToLocal(path, *, overwrite=False)#: Save Estimator / Transformer / Model / Evaluator to provided local path.

New in version 3.5.0.

set(param, value)#: Sets a parameter in the embedded param map.

setCollectSubModels(value)[source]#: Sets the value of collectSubModels.

setEstimator(value)[source]#: Sets the value of estimator.

New in version 3.5.0.

setEstimatorParamMaps(value)[source]#: Sets the value of estimatorParamMaps.

New in version 3.5.0.

setEvaluator(value)[source]#: Sets the value of evaluator.

New in version 3.5.0.

setFoldCol(value)[source]#: Sets the value of foldCol.

New in version 3.5.0.

setNumFolds(value)[source]#: Sets the value of numFolds.

New in version 3.5.0.

setParallelism(value)[source]#: Sets the value of parallelism.

setParams(*, estimator=None, estimatorParamMaps=None, evaluator=None, numFolds=3, seed=None, parallelism=1, foldCol='')[source]#: setParams(self, *, estimator=None, estimatorParamMaps=None, evaluator=None, numFolds=3, seed=None, parallelism=1, collectSubModels=False, foldCol=””): Sets params for cross validator.

New in version 3.5.0.

setSeed(value)[source]#: Sets the value of seed.

Attributes Documentation

estimator = Param(parent='undefined', name='estimator', doc='estimator to be cross-validated')#

estimatorParamMaps = Param(parent='undefined', name='estimatorParamMaps', doc='estimator param maps')#

evaluator = Param(parent='undefined', name='evaluator', doc='evaluator used to select hyper-parameters that maximize the validator metric')#

foldCol = Param(parent='undefined', name='foldCol', doc="Param for the column name of user specified fold number. Once this is specified, :py:class:`CrossValidator` won't do random k-fold split. Note that this column should be integer type with range [0, numFolds) and Spark will throw exception on out-of-range fold numbers.")#

numFolds = Param(parent='undefined', name='numFolds', doc='number of folds for cross validation')#

parallelism = Param(parent='undefined', name='parallelism', doc='the number of threads to use when running parallel algorithms (>= 1).')#

params#: Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.

seed = Param(parent='undefined', name='seed', doc='random seed.')#

uid#: A unique id for the object.