pyspark.sql.functions.overlay#

pyspark.sql.functions.overlay(src, replace, pos, len=- 1)[source]#

Overlay the specified portion of src with replace, starting from byte position pos of src and proceeding for len bytes.

New in version 3.0.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

srcColumn or column name: the string that will be replaced
replaceColumn or column name: the substitution string
posColumn or column name or int: the starting position in src
lenColumn or column name or int, optional: the number of bytes to replace in src string by ‘replace’ defaults to -1, which represents the length of the ‘replace’ string

Returns

Column: string with replaced values.

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("SPARK_SQL", "CORE")], ("x", "y"))
>>> df.select("*", sf.overlay("x", df.y, 7)).show()
+---------+----+--------------------+
|        x|   y|overlay(x, y, 7, -1)|
+---------+----+--------------------+
|SPARK_SQL|CORE|          SPARK_CORE|
+---------+----+--------------------+

>>> df.select("*", sf.overlay("x", df.y, 7, 0)).show()
+---------+----+-------------------+
|        x|   y|overlay(x, y, 7, 0)|
+---------+----+-------------------+
|SPARK_SQL|CORE|      SPARK_CORESQL|
+---------+----+-------------------+

>>> df.select("*", sf.overlay("x", "y", 7, 2)).show()
+---------+----+-------------------+
|        x|   y|overlay(x, y, 7, 2)|
+---------+----+-------------------+
|SPARK_SQL|CORE|        SPARK_COREL|
+---------+----+-------------------+