26,327 questions
0
votes
0
answers
20
views
Issue with Missing Tables During Google Ads Data Backfill in BigQuery
We are currently extracting Google Ads data into BigQuery. However, we encountered errors on certain scheduled dates, so we attempted a backfill. It seems that some tables were not populated for those ...
- 1
0
votes
1
answer
64
views
Calculating AVERAGE for past dates from latest date
I have a dataset as shown below. The requirement is to write a SQL query that would :
calculate the average rec_cnt of the previous 3 dates pertaining to
the latest date for a specific tbl_nm
apply +...
- 825
0
votes
0
answers
23
views
BigQuery select last modified time using __TABLES__ from all tables in all datasets?
I would like to get list or csv and store it in GCS, where the list contains list of all the tables and the last modified date.
I have more than 130 datasets and 10-20 tables in each datasets. Since I ...
- 1
0
votes
1
answer
31
views
Cheapest way to check if two BigQuery tables are identical up to row ordering?
I need to check if two BigQuery tables are identical, where the only difference might be the ordering of rows. Obviously, I want to avoid scanning both tables completely.
I initially thought of ...
- 143
0
votes
0
answers
36
views
How to prevent deletions from source (GCP CloudSQL MySQL) reflecting in GCP BigQuery using Datastream?
Description:
We are currently using Google Cloud Datastream to replicate data from a CloudSQL (MySQL) instance into BigQuery in near real-time. The replication works perfectly for insert and update ...
1
vote
1
answer
42
views
Seeing mixed results from clustering in BigQuery
We have tables partitioned by day, and we want to additionally improve with clustering.
I understand from How can I improve the amount of data queried with a partitioned+clustered table? that ...
- 33
1
vote
4
answers
50
views
Implementing iteration in Standard SQL (BigQuery)
I have an audit table (with tablename, date and rec counts) with data as far back as 5 yrs as shown below. My requirement is to retrieve the record counts for the tablename for a specific day and it's ...
- 825
0
votes
0
answers
32
views
Missing some events in BigQuery from Firebase Analytics integration
For some reason, certain events that are normally visible in Firebase Analytics cannot be found in our BigQuery database. Specifically, this concerns the screen_view and user_engagement events. These ...
- 730
0
votes
0
answers
23
views
GBQ region mismatch error when using Google Ads Transfer data US in Dataform repo
I'm working on a reporting pipeline using Dataform in GCP to build Google Ads dashboards - Client Dashboard for each client and an Agency Internal Dashboard.
Here's my setup:
My Google Ads Transfer ...
- 405
0
votes
0
answers
41
views
+50
BigQuery Performance Issue After Switching Data Pipeline to DataFlow
Problem
I'm experiencing significant query performance degradation in BigQuery for recent partitions after switching our data pipeline from a sequential Talend approach to Apache Beam/DataFlow.
...
- 143
0
votes
1
answer
44
views
Error Exporting from BigQuery to GCS: 'Operation cannot be performed on a nested schema
I encountered an error while trying to execute an ETL task to export data from a BigQuery table to Google Cloud Storage (GCS).
Here is the exact error message:
raise self._exception google.api_core....
1
vote
0
answers
44
views
Stop my temporary table from filtering out data unprompted
I'm aggregating US county cost of living data, but I realized my temporary table is only returning rows for families without kids for some reason. For every county, there's a row for families from no ...
- 11
0
votes
0
answers
19
views
Permission fro streaming data from firebase to bigquery
I'm using the Firestore to BigQuery streaming extension to connect Firestore with BigQuery. During setup, the extension automatically created a dedicated service account separate from my main project ...
0
votes
1
answer
55
views
BigQuery aggregations on different level by grouping sets
I need to create an aggregation for each housing type (column type) for each destination.
There is a hierarchy (product->sub_product) and (category->sub_category) and also product can have ...
- 19
0
votes
0
answers
42
views
Pyspark not inserting data into BigQuery
I have a PySpark code which reads data from some bigquery external tables and inserts into bigquery native tables.
By using indirect mode of insertion, the flow should be as such.
PySpark reads from ...
- 413