3

My particular problem that I'd like to solve is a table with very wide column (sometimes > 1000 columns) which requires me to split the table. In order to do so smartly, I'd like to split right before I get Row size too large (> 8126) or The maximum row size for the used table type, not counting BLOBs, is 65535..

For that purpose, I'm trying to find the actual size for each columns. Some are easy (like BOOLEAN, INT, DATE, TEXT, etc). I'm trying to figure out VARCHAR. Hence the experiment of creating tables with many VARCHAR columns.

However, I cannot really make sense of it. I tried experimenting with a table containing only VARCHAR(217), I was able to create 100 columns. The next column (101th) maximum size is VARCHAR(78).

CREATE TABLE IF NOT EXISTS test
(
col1    VARCHAR(14) NOT NULL,
col2    VARCHAR(14) NOT NULL,
...
col100  VARCHAR(217) NOT NULL,
col101  VARCHAR(78) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

I figured out that the reasoning is as below:

  1. 2 bytes for storage of length value
  2. 217x3 = 651 bytes for off-page storage
  3. So 100 columns = 2x100 + 651x100 = 65300
  4. Max is 65535 - 65300 = 235 bytes
  5. Therefore, we can fit 1 more column of maximum (235 - 2) / 3 = 77.666666666666667 … or VARCHAR(78)

Applying the above for VARCHAR(128), I expect 65535 / (128x3+2) or 169 columns with the last column can fit VARCHAR(99). And it is indeed, true. This works:

CREATE TABLE IF NOT EXISTS test
(
col1    VARCHAR(128) NOT NULL,
col2    VARCHAR(128) NOT NULL,
...
col169  VARCHAR(128) NOT NULL,
col170  VARCHAR(99) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

Applying the same logic for VARCHAR(64) doesn't work, however. Applying the same logic above, I would expect 337 columns. However, I can only get 197 columns. Anything more than that, I got Row size too large (> 8126) error.

CREATE TABLE IF NOT EXISTS test
(
col1    VARCHAR(64) NOT NULL,
col2    VARCHAR(64) NOT NULL,
...
col196  VARCHAR(64) NOT NULL,
col197  VARCHAR(64) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

The same thing is for VARCHAR(14). I can also create 197 columns max before hitting Row size too large (> 8126) error.

MySQL documentation says

When a table is created with ROW_FORMAT=DYNAMIC, InnoDB can store long variable-length column values (for VARCHAR, VARBINARY, and BLOB and TEXT types) fully off-page, with the clustered index record containing only a 20-byte pointer to the overflow page.

which I suspect is what's going on here where InnoDB would store some values fully off page because there's no way 197 of 64-length utf8 strings can fit in 8126. If all of the 64-length utf8 strings are stored in-line, I would expect 41 columns maximum ( 8126 / (64x3+2) ).

Any help appreciated. I'm using MySQL 5.7.26-29, Barracuda file format and 16K page size.

1
  • This is a dup of a similar Question in stackoverflow. Commented Jul 14, 2021 at 14:26

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.