All Questions
Tagged with database-partitioning google-bigquery
26 questions
0
votes
0
answers
58
views
Non-uniform integer partitioning in Google BigQuery
In BigQuery integer range partitioning, is it possible to create non-uniform partition intervals? The use case that I have is that the date column is an integer (20201231 for example). All current ...
- 1,102
0
votes
0
answers
41
views
Best way to create half-yearly partitions in Google Big Query
Big Query supports partitioning a time-unit column based on daily, hourly, monthly or yearly granularity. However, I would like to partition based on half-yearly granularity which is not directly ...
- 1,102
4
votes
0
answers
734
views
Big Query : Materialized view getting invalid after CREATE or REPLACE script of Partitioned Base table
I have a Partitioned base table my_project.my_dataset.base_table with a usage_start_time column ( used to partition this base table). And I also have a materialized view which references this base ...
- 207
1
vote
0
answers
374
views
Bigquery SELECT query cost much higher than table size
I have a BigQuery table which is filled by a datastream from CloudSQL. Recently I tried partitioning the table by DAY to reduce query costs. This didn't work (partition pruning was not being applied)...
0
votes
1
answer
53
views
GCP BigQuery wont access a single partition
I'm experiencing a problem with a GBQ query which goes through all the data in a table, even though the table is partitioned.
The partitioning is set to DAY on the datetime field.
Currently the query ...
- 177
0
votes
2
answers
284
views
Bigquery estimates (and bills) query cost incorrectly in partitioned table
I have a table similar to the following
CREATE TABLE `main.viewings`
(
event_time TIMESTAMP,
other_columns INT64,
more_columns STRING
)
PARTITION BY DATE(event_time)
OPTIONS(
...
- 2,315
0
votes
1
answer
382
views
Issue creating table partition by date with data from other table. In Google Cloud
I'm doing a course from Google. Modernizing Data Lakes and Data Warehouses. And in the quest they ask to create a partition table and populate all the info from the source table, I execute my code, ...
- 56
0
votes
1
answer
267
views
Unioned table costs more to query in BigQuery than individual tables?
BigQuery cost scenarios
When I query a large unioned table - partitioned by date field and clustered by a clientkey field - for a specific client's data it appears to process more data than if I just ...
0
votes
1
answer
2k
views
Prune BigQuery partitions when building incremental table with DBT
I have the same problem again as this question:
How to choose the latest partition in BigQuery?
What's the problem?
How do you make an incremental table using BigQuery in DBT without scanning the ...
- 839
0
votes
1
answer
1k
views
Find first event occurring after given event
I am working with a table consisting of a number of web sessions with various events and event id:s. To simplify my question, let's say that I have 4 columns which are session_id, event_name and ...
- 43
0
votes
1
answer
339
views
BigQuery partition pruning while using analytical function as part of a view, on a DATETIME field with DAY granularity
I am trying to use analytical functions (e.g. FIRST_VALUE), while still benefiting from from partition pruning. This while the table is partitioned on a DATETIME field with DAY granularity.
Example ...
- 3,134
1
vote
1
answer
45
views
How to partition an existing 120 TB table in bigquery?
I have a 120 TB table which unfortunately was not partitioned on creation. The jobs that use it filter it by a date column called sales_date and are taking hours to complete. I am trying to find the ...
1
vote
2
answers
705
views
How to use partition pruning in google big query without hardcoding the filter condition?
I have a big table in GBQ which is partitioned by date. and I want to use partition pruning to optimize my query. the problem is that filter condition is a value that is read from another table and I ...
- 587
2
votes
0
answers
715
views
BigQuery supposedly hourly partitioned tables show up as different tables
When I push data to BQ using pandas (to_gbq()) the ones which are supposed to be hourly partitioned tables, show up as different tables with same naming convention e.g.:
The naming convention I made ...
- 6,085
-1
votes
1
answer
34
views
Creating a partitioned table from query in Big Query does not yield same as without partitioning
When creating a table let's say "orders" with partitioning in the following way my result gets truncated in comparison to if I create it without partitioning. (Commenting and uncommenting ...
