Skip to content

Commit 3660c4d

Browse files
Pushap GoyalHerman Lee
Pushap Goyal
authored and
Herman Lee
committed
Handle large write batch for intrinsic tmp tables
Summary: If the intrinsic tmp tables goes beyond some threshold, then we commit the write batch accumulated so far and start new transaction handling the write batch. **Changes** * Added a variable rocksdb_max_intrinsic_tmp_table_write_count which controls the size of write batch allowed for intrinsic tmp table. Once we cross that threshold, we will commit the current transaction and then start a new transaction. * If there are valid open iterators before we commit intrinsic tmp table transaction, then we refresh all iterators back to previous position after committing transaction. Recursive cte opens multiple iterators for same transaction and we need to refresh all iterators after we commit the current write batch for transaction. * I also refreshed the iterators after the first write to the write batch. This is done to handle the special case where we start rocksdb iterator on new transaction(with empty write batch). Then iterator will only see the already committed data, but ignores any new data added in write batch later. So we are pro-actively refresh the iterator after first write in write batch. * Added a status variable rocksdb_intrinsic_tmp_table_commits to count number of intrinsic tmp table commits. * create_ondisk_from_heap method uses table->record[1] for insertion. Myrocks encode_value_slice relies on table->record[0] for value encoding. We have reposition the fields in write_row specifically for intrinsic tmp table to point to record[1]. Better way to fix this is to remove the dependency on record[1] from encode_value_slice method and add dependency on input buf from write_row. Reviewed By: luqun Differential Revision: D37835016 --------------------------------------------------------------------------------------- Temporarily disable some of rocksdb.tmp_table test Summary: RocksDB tmp table needs bug fixes, and meanwhile disabling some of the rocksdb.tmp_table tests so that it won't report failures. Reviewed By: lth Differential Revision: D41391615 --------------------------------------------------------------------------------------- set m_read_opts.ignore_range_deletions properly (facebook#1218) Summary: ignore_range_deletions should be set to true when range del is not enabled Pull Request resolved: facebook#1218 Reviewed By: Pushapgl Differential Revision: D39115879 Pulled By: yoshinorim
1 parent 37535d6 commit 3660c4d

16 files changed

Lines changed: 443 additions & 166 deletions

‎mysql-test/combinations‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
[innodb_intrinsic_table]
22
enable_rocksdb_intrinsic_tmp_table= OFF
3-
loose-rocksdb_enable_tmp_table = OFF
3+
loose-rocksdb_enable_tmp_table= OFF
44

55
[rocksdb_intrinsic_table]
66
enable_rocksdb_intrinsic_tmp_table= ON
7-
loose-rocksdb_enable_tmp_table = ON
8-
loose-rocksdb_enable_delete_range_for_drop_index = ON
7+
loose-rocksdb_enable_tmp_table= ON
8+
loose-rocksdb_enable_delete_range_for_drop_index= ON
9+
loose-rocksdb_max_intrinsic_tmp_table_write_count= 3

‎mysql-test/r/mysqld--help-notwin.result‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2196,6 +2196,10 @@ The following options may be given as the first argument:
21962196
Track history for at most this many completed
21972197
compactions. The history is in the
21982198
INFORMATION_SCHEMA.ROCKSDB_COMPACTION_HISTORY table.
2199+
--rocksdb-max-intrinsic-tmp-table-write-count=#
2200+
Intrinsic tmp table max allowed write batch size.After
2201+
this, current transaction holding write batch will commit
2202+
and newtransaction will be started.
21992203
--rocksdb-max-latest-deadlocks=#
22002204
Maximum number of recent deadlocks to store
22012205
--rocksdb-max-log-file-size=#
@@ -3652,6 +3656,7 @@ rocksdb-max-background-flushes -1
36523656
rocksdb-max-background-jobs 2
36533657
rocksdb-max-bottom-pri-background-compactions 0
36543658
rocksdb-max-compaction-history 64
3659+
rocksdb-max-intrinsic-tmp-table-write-count 1000
36553660
rocksdb-max-latest-deadlocks 5
36563661
rocksdb-max-log-file-size 0
36573662
rocksdb-max-manifest-file-size 1073741824

‎mysql-test/suite/json/r/json_table.result‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1033,6 +1033,7 @@ Variable_name Value
10331033
Created_tmp_disk_tables 1
10341034
Created_tmp_files 0
10351035
Created_tmp_tables 1
1036+
rocksdb_intrinsic_tmp_table_commits 0
10361037
set @@max_heap_table_size= @save_heap_size;
10371038
set @@internal_tmp_mem_storage_engine= @save_mem_se;
10381039
#
@@ -1051,6 +1052,7 @@ Variable_name Value
10511052
Created_tmp_disk_tables 0
10521053
Created_tmp_files 0
10531054
Created_tmp_tables 1
1055+
rocksdb_intrinsic_tmp_table_commits 0
10541056
#
10551057
# Bug#25525409: ASSERTION `TABLE_LIST->TABLE' FAILED IN SQL/SQL_BASE.CC
10561058
#

‎mysql-test/suite/rocksdb/r/rocksdb.result‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -999,6 +999,7 @@ rocksdb_max_background_flushes -1
999999
rocksdb_max_background_jobs 2
10001000
rocksdb_max_bottom_pri_background_compactions 0
10011001
rocksdb_max_compaction_history 64
1002+
rocksdb_max_intrinsic_tmp_table_write_count 1000
10021003
rocksdb_max_latest_deadlocks 5
10031004
rocksdb_max_log_file_size 0
10041005
rocksdb_max_manifest_file_size 1073741824
@@ -1600,6 +1601,7 @@ rocksdb_table_index_stats_success #
16001601
rocksdb_table_index_stats_failure #
16011602
rocksdb_table_index_stats_req_queue_length #
16021603
rocksdb_covered_secondary_key_lookups #
1604+
rocksdb_intrinsic_tmp_table_commits #
16031605
rocksdb_additional_compaction_triggers #
16041606
rocksdb_binlog_ttl_compaction_timestamp #
16051607
rocksdb_block_cache_add #
@@ -1754,6 +1756,7 @@ ROCKSDB_GET_HIT_L1
17541756
ROCKSDB_GET_HIT_L2_AND_UP
17551757
ROCKSDB_GIT_DATE
17561758
ROCKSDB_GIT_HASH
1759+
ROCKSDB_INTRINSIC_TMP_TABLE_COMMITS
17571760
ROCKSDB_ITER_BYTES_READ
17581761
ROCKSDB_LAST_LEVEL_SEEK_DATA
17591762
ROCKSDB_LAST_LEVEL_SEEK_DATA_USEFUL_FILTER_MATCH
@@ -1889,6 +1892,7 @@ ROCKSDB_GET_HIT_L1
18891892
ROCKSDB_GET_HIT_L2_AND_UP
18901893
ROCKSDB_GIT_DATE
18911894
ROCKSDB_GIT_HASH
1895+
ROCKSDB_INTRINSIC_TMP_TABLE_COMMITS
18921896
ROCKSDB_ITER_BYTES_READ
18931897
ROCKSDB_LAST_LEVEL_SEEK_DATA
18941898
ROCKSDB_LAST_LEVEL_SEEK_DATA_USEFUL_FILTER_MATCH

‎mysql-test/suite/rocksdb/t/tmp_table.test‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,3 +355,21 @@ set session debug="-d,crash_during_write_row";
355355
show create table tmp1;
356356
select * from t1;
357357
drop table t1;
358+
359+
####################################################
360+
### [TODO] Fix bugs: Case 12: Intrinsic tmp table background commits
361+
####################################################
362+
#--echo Case 12: Intrinsic tmp table background commits
363+
#set big_tables = 1;
364+
#Set global rocksdb_max_intrinsic_tmp_table_write_count = 2;
365+
#let $old_value= query_get_value(show status like "rocksdb_intrinsic_tmp_table_commits", Value, 1);
366+
#create table t1(i int, c char(5));
367+
#insert into t1 values (0, "aaaa");
368+
#insert into t1 values (1, "aaaa");
369+
#select i, c, count(*) from t1 group by i, c having count(*) > 0;
370+
#let $new_value= query_get_value(show status like "rocksdb_intrinsic_tmp_table_commits", Value, 1);
371+
#--let $assert_text = Number of intrinsic tmp table commits should be 1
372+
#--let $assert_cond= $new_value - $old_value = 1
373+
#--source include/assert.inc
374+
#Set global rocksdb_max_intrinsic_tmp_table_write_count = default;
375+
#drop table t1;

‎mysql-test/suite/rocksdb_sys_vars/r/rocksdb_max_intrinsic_tmp_table_write_count_basic.result‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
CREATE TABLE valid_values (value varchar(255)) ENGINE=myisam;
2+
INSERT INTO valid_values VALUES(10);
3+
INSERT INTO valid_values VALUES(20);
4+
CREATE TABLE invalid_values (value varchar(255)) ENGINE=myisam;
5+
INSERT INTO invalid_values VALUES('\'aaa\'');
6+
SET @start_global_value = @@global.rocksdb_max_intrinsic_tmp_table_write_count;
7+
SELECT @start_global_value;
8+
@start_global_value
9+
1000
10+
'# Setting to valid values in global scope#'
11+
"Trying to set variable @@global.rocksdb_max_intrinsic_tmp_table_write_count to 10"
12+
SET @@global.rocksdb_max_intrinsic_tmp_table_write_count = 10;
13+
SELECT @@global.rocksdb_max_intrinsic_tmp_table_write_count;
14+
@@global.rocksdb_max_intrinsic_tmp_table_write_count
15+
10
16+
"Setting the global scope variable back to default"
17+
SET @@global.rocksdb_max_intrinsic_tmp_table_write_count = DEFAULT;
18+
SELECT @@global.rocksdb_max_intrinsic_tmp_table_write_count;
19+
@@global.rocksdb_max_intrinsic_tmp_table_write_count
20+
1000
21+
"Trying to set variable @@global.rocksdb_max_intrinsic_tmp_table_write_count to 20"
22+
SET @@global.rocksdb_max_intrinsic_tmp_table_write_count = 20;
23+
SELECT @@global.rocksdb_max_intrinsic_tmp_table_write_count;
24+
@@global.rocksdb_max_intrinsic_tmp_table_write_count
25+
20
26+
"Setting the global scope variable back to default"
27+
SET @@global.rocksdb_max_intrinsic_tmp_table_write_count = DEFAULT;
28+
SELECT @@global.rocksdb_max_intrinsic_tmp_table_write_count;
29+
@@global.rocksdb_max_intrinsic_tmp_table_write_count
30+
1000
31+
"Trying to set variable @@session.rocksdb_max_intrinsic_tmp_table_write_count to 444. It should fail because it is not session."
32+
SET @@session.rocksdb_max_intrinsic_tmp_table_write_count = 444;
33+
ERROR HY000: Variable 'rocksdb_max_intrinsic_tmp_table_write_count' is a GLOBAL variable and should be set with SET GLOBAL
34+
'# Testing with invalid values in global scope #'
35+
"Trying to set variable @@global.rocksdb_max_intrinsic_tmp_table_write_count to 'aaa'"
36+
SET @@global.rocksdb_max_intrinsic_tmp_table_write_count = 'aaa';
37+
Got one of the listed errors
38+
SELECT @@global.rocksdb_max_intrinsic_tmp_table_write_count;
39+
@@global.rocksdb_max_intrinsic_tmp_table_write_count
40+
1000
41+
SET @@global.rocksdb_max_intrinsic_tmp_table_write_count = @start_global_value;
42+
SELECT @@global.rocksdb_max_intrinsic_tmp_table_write_count;
43+
@@global.rocksdb_max_intrinsic_tmp_table_write_count
44+
1000
45+
DROP TABLE valid_values;
46+
DROP TABLE invalid_values;

‎mysql-test/suite/rocksdb_sys_vars/t/rocksdb_max_intrinsic_tmp_table_write_count_basic.test‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
--source include/have_rocksdb.inc
2+
3+
CREATE TABLE valid_values (value varchar(255)) ENGINE=myisam;
4+
INSERT INTO valid_values VALUES(10);
5+
INSERT INTO valid_values VALUES(20);
6+
7+
CREATE TABLE invalid_values (value varchar(255)) ENGINE=myisam;
8+
INSERT INTO invalid_values VALUES('\'aaa\'');
9+
10+
--let $sys_var=rocksdb_max_intrinsic_tmp_table_write_count
11+
--let $read_only=0
12+
--let $session=0
13+
--source ../include/rocksdb_sys_var.inc
14+
15+
DROP TABLE valid_values;
16+
DROP TABLE invalid_values;

‎mysql-test/t/error_simulation.test‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
-- source include/have_debug.inc
2-
--source include/have_innodb_intrinsic_table.inc
3-
42
#
53
# Bug #28499: crash for grouping query when tmp_table_size is too small
64
#

‎mysql-test/t/with_recursive_innodb_tmp_table.test‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
--source include/have_innodb_intrinsic_table.inc
21
--source include/have_64bit.inc
32
--source include/no_valgrind_without_big.inc
43

‎mysql-test/t/with_recursive_rocksdb-master.opt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
--loose-rocksdb_max_intrinsic_tmp_table_write_count=1000000

0 commit comments

Comments
 (0)