numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Question

I want to call my Python module from the Matlab. I received the error:

Error using numpy_ops>init thinc.backends.numpy_ops

Python Error:

 ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject.

The Python script is as follows

import spacy
def text_recognizer(model_path, text):
try:
    # Load the trained model
    nlp = spacy.load(model_path)
    print("Model loaded successfully.")
    
    # Process the given text
    doc = nlp(text)
    ent_labels = [(ent.text, ent.label_) for ent in doc.ents]
        return ent_labels

The Matlab script is as follows

% Set up the Python environment
pe = pyenv;
py.importlib.import_module('final_output');

% Add the directory containing the Python script to the Python path
path_add = fileparts(which('final_output.py'));
if count(py.sys.path, path_add) == 0
    insert(py.sys.path, int64(0), path_add);
end
% Define model path and text to process
model_path = 'D:\trained_model\\output\\model-best';
text = 'Roses are red';
% Call the Python function
pyOut = py.final_output.text_recognizer(model_path, text);
% Convert the output to a MATLAB cell array
entity_labels = cell(pyOut);
disp(entity_labels);

I found one solution to update Numpy, what I did, but nothing changed. I am using Python 3.9 and Numpy version 2.0.0

The error was received when I tried to call the Python module using a Matlab script.

How can I fix the issue?

Answer 1

The reason is that pandas defines its numpy dependency freely as "anything newer than certain version of numpy". The problem occured, when numpy==2.0.0 has been released on June 16th 2024, because it is no longer compatible with your pandas version.

The solution is to pin down the numpy version to any before the 2.0.0. Today it could be (this is the most recent numpy 1 release):

numpy==1.26.4

To be added in your requirements or to the pip command you use (but together with installing pandas).

Nowadays pip is very flexible and can handle the issue flawesly. You just need to ask it to install both pandas and numpy of given versions in the same pip install invocation.

Answer 2

downgrade your numpy version to 1.26.4

Answer 3

We encountered this same problem on our prod.
We did not specify a version of numpy in our requirements.txt.

So the solution was to add the numpy==1.26.3 in the file so that pandas uses that version instead.

Solution
Add numpy==1.26.3 in your requirements.txt.

Answer 4

11.02.2025 Update

Starting with Pandas 2.2.0, it is now compatible with Numpy >= 2.

https://github.com/pandas-dev/pandas/issues/55519#issuecomment-1867095954

pip install pandas==2.2.0

or

pip install --upgrade pandas

Answer 5

Just had some troubles with my airflow cicd, fixed including numpy 1.23.2 version on my requeriments, before pandas.

Answer 6

The alternative solution is to upgrade your panda's library, which will automatically downgrade your numpy and fix any other compatibility issues.

pip install --upgrade pandas pandasai

after running this you can check your versions and see that it automatically picked 1.24.4 instead of the one I had which was 2.0.2.

print(f"NumPy version: {np.__version__}")
print(f"Pandas version: {pd.__version__}")

Output:

NumPy version: 1.24.4
Pandas version: 1.5.3

Note: This might uninstall some libraries that might cause compatibility problems with pandas later also check the update in the terminal.

Answer 7

If you use Google Colab, currently downgrading numpy to 1.26 will still give you some errors like you were still using numpy 2.0, so e.g. also:

"No module named 'numpy.strings'"

or

"ModuleNotFoundError: No module named 'numpy.rec'"

The Colab doesn't prompt you to restart the environment in order to use numpy 1.26 after downgrade. You need to do that manually. So go to Environment, and launch the session again (so don't delete it).

Answer 8

A cleanup before re-installation saves me. Credit to this comment:

site_packages=$(python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
rm -rf "$site_packages"/numpy
pip install --upgrade --force-reinstall numpy

However, this did not work until I changed the second line to numpy*, which removed all folders that start with numpy. I noticed there are other NumPy-related folders like numpy-2.0.0.dist-info/ under the $site_packages.