Spark: convert array of arrays to array of strings?

Question

I have column with type Array of Arrays I need to get column array of string.

+--------------------------+
|field                     |
+--------------------------+
|[[1, 2, 3], [1, 2, 3], []]|
+--------------------------+

I need to get:

+--------------------------+
|field                     |
+--------------------------+
|["123", "123", ""]        |
+--------------------------+

Can I do this in Spark without using UDF?

Mohana B C · Accepted Answer · 2023-02-04 17:05:09Z

2

You can use transform higher order function,

import spark.implicits._

val df = Seq(Seq(Seq(1,2,3), Seq(1,2,3), Seq())).toDF("field")

df.withColumn("field", expr("transform(field, v->concat_ws('',v))"))
    .show

+------------+
|       field|
+------------+
|[123, 123, ]|
+------------+

answered Feb 4, 2023 at 17:05

Mohana B C

5,4721 gold badge13 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alexander Lopatin Over a year ago

That's great! I couldn't find this function. Do you know how I can return array without 1st element? I found how to get first element, but I can't find how to get from this [1, 2, 3] this [2,3]

Mohana B C Over a year ago

You can use transform function in this way - transform(field, v->slice(v, 2, size(v)))

Collectives™ on Stack Overflow

Spark: convert array of arrays to array of strings?

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related