When loading a very large gctx file ~24Gb on my laptop with 16 Gb using the function cmapPy.pandasGEXpress.parse.parse, I run out of memory with the following error:
Unable to allocate array with shape (473647,) and data type
If I use cidx to select a very low number of columns, then there is no more error.
However, when I request certain columns and certain rows, using both cidx and ridx, the same allocation error occurs. This indicates that the row filtering is applied, followed by the column filtering. This is a bad behaviour when dealing with very large cmap files, where it would be preferable that both filtering be applied simultaneously to avoid running out of RAM.
The problem comes from pandasGEXpress.parse_metadata_df, at the line curr_dset.read_direct(temp_array).
The function read_direct simply reads all the rows/columns without any means of filtering.
When loading a very large gctx file ~24Gb on my laptop with 16 Gb using the function
cmapPy.pandasGEXpress.parse.parse, I run out of memory with the following error:Unable to allocate array with shape (473647,) and data typeIf I use
cidxto select a very low number of columns, then there is no more error.However, when I request certain columns and certain rows, using both
cidxandridx, the same allocation error occurs. This indicates that the row filtering is applied, followed by the column filtering. This is a bad behaviour when dealing with very large cmap files, where it would be preferable that both filtering be applied simultaneously to avoid running out of RAM.The problem comes from
pandasGEXpress.parse_metadata_df, at the linecurr_dset.read_direct(temp_array).The function
read_directsimply reads all the rows/columns without any means of filtering.