GCP Dataflow Error: Path "gs://..." is not a valid filepattern. The pattern must be of the form "gs://<bucket>/path/to/file"

Question

I am trying to create a dataflow from Pub-Sub to BigQuery in GCP console.

In the "Create job from template" screen, I am having a trouble what to enter for "Temporary Location" box. It says "Path and filename prefix for writing temporary files. ex: gs://MyBucket/tmp".

So I specified something like this: "gs://${GOOGLE_CLOUD_PROJECT}-test/dataflow/tmp"

But I am getting this error (dataflow folder is there BTW):

Path "gs://${GOOGLE_CLOUD_PROJECT}-test/dataflow/tmp" is not a valid  filepattern. The pattern must be of the form "gs://<bucket>/path/to/file".

I tried different patterns but to no avail. Any idea how to resolve this?

Martin Zeitler · Accepted Answer · 2018-08-10 23:02:13Z

2

it complains that it wants a bucket ...

The pattern must be of the form "gs:// [bucket] /path/to/file".

export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
export BUCKET_NAME="${PROJECT_ID}-test"

gsutil "gs://${BUCKET_NAME}/dataflow/tmp"

wondered about the -test suffix and I've just tried to reflect that in code.

one can obtain all valid BUCKET_NAMEs with gsutil ls.

edited Aug 10, 2018 at 23:02

answered Aug 10, 2018 at 22:38

Martin Zeitler

77.3k20 gold badges166 silver badges239 bronze badges

That's a valid point. The bucket named "${GOOGLE_CLOUD_PROJECT}-test" is there but then I realized that this macro is not recognized there different from what this doc says (codelabs.developers.google.com/codelabs/…). When I entered the bucket name itself without the macro, it worked! Thanks!
– kee
Commented Aug 10, 2018 at 22:57

Add a comment |

Collectives™ on Stack Overflow

GCP Dataflow Error: Path "gs://..." is not a valid filepattern. The pattern must be of the form "gs://<bucket>/path/to/file"

1 Answer 1

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Related