如何使用 pyspark 更新结构体嵌套列中的值

我尝试做非常简单的事情 - 更新嵌套列的值；但是，我不知道如何

环境：

1. 阿帕奇火花2.4.5
2. 数据块 6.4
3. Python 3.7

dataDF = [
  (('Jon','','Smith'),'1580-01-06','M',3000)
]


schema = StructType([
        StructField('name', StructType([
             StructField('firstname', StringType(), True),
             StructField('middlename', StringType(), True),
             StructField('lastname', StringType(), True)
             ])),
         StructField('dob', StringType(), True),
         StructField('gender', StringType(), True),
         StructField('gender', IntegerType(), True)
         ])


df = spark.createDataFrame(data = dataDF, schema = schema)
df = df.withColumn("name.firstname", lit('John'))
df.printSchema()
df.show()

#Results
#I get a new column instead of update

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- dob: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- gender: integer (nullable = true)
 |-- name.firstname: string (nullable = false)

+--------------+----------+------+------+--------------+
|          name|       dob|gender|gender|name.firstname|
+--------------+----------+------+------+--------------+
|[Jon, , Smith]|1580-01-06|     M|  3000|          John|
+--------------+----------+------+------+--------------+

对于 Spark 3.1+，您可以使用与字段在结构列上：

添加/替换字段的表达式StructType按名字。

import pyspark.sql.functions as F

df1 = df.withColumn("name", F.col("name").withField("firstname", F.lit("John")))