mickdelaney
1/8/2017 - 8:13 PM

Writing an UDF for withColumn in PySpark

Writing an UDF for withColumn in PySpark

from pyspark.sql.types import StringType
from pyspark.sql.functions import udf

maturity_udf = udf(lambda age: "adult" if age >=18 else "child", StringType())

df = sqlContext.createDataFrame([{'name': 'Alice', 'age': 1}])
df.withColumn("maturity", maturity_udf(df.age))