1

buckets is a column of type array<string>. The logic is similar to array_intersect, except only the prefix of each string in buckets (before the first -) is compared. How can I optimize the following clause?

transform(
    a.buckets, -- buckets is a set of strings
    x -> if(
        array_contains(c.names, split(x, '-')[0]), -- c is a CTE, names is an array/set of strings
        x,
        null
    )
) AS buckets

There is also a case where bucket is a single string column:

transform(
    split(bucket, ','), -- bucket is a string
    x -> if(
        array_contains(c.names, split(x, '-')[0]), -- c is a CTE, names is an array/set of strings
        x,
        null
    )
) AS buckets

How can this prefix-based filtering be optimized in Hive SQL executed by the Spark engine?

Improve this question
Related questions
1
How to set the execution engine to spark when accessing Cloudera Hive via JDBC
0
How to add new column in to partition by clause in Hive External Table
1
Explode array with nested array raw spark sql
Load 7 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.