I am trying to read a CSV file using the below script.
Past = pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv", encoding='utf-8')
But, I get the error "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 35: invalid start byte"
Where is the issue?
I used encoding in the script and thought it would resolve the error.
This happens because you chose the wrong encoding.
Since you are working on a Windows machine, just replacing
Past = pd.read_csv("C:/Users/.../Past.csv", encoding='utf-8')
with
Past = pd.read_csv("C:/Users/.../Past.csv", encoding='cp1252')
should solve the problem.
cp1252 was the proper encoding? Chances are it wasn't but you got lucky because it stopped throwing errors, but now you have incorrect characters in your data.
Try using :
pd.read_csv("Your filename", encoding="ISO-8859-1")
The code that I parsed from some website was converted in this encoding instead of default UTF-8 encoding which is standard.
ISO-8859-1 eliminates all the errors because every possible byte maps to a valid character. Doesn't mean the characters are correct though. How did you determine the correct encoding used by the website?Use this solution. It will strip out (ignore) the characters and return the string without them. Only use this if your need is to strip them, not convert them.
with open(path, encoding="utf8", errors='ignore') as f:
Using errors='ignore', you’ll just lose some characters. But if you don't care about them as they seem to be extra characters originating from a the bad formatting and programming of the clients connecting to my socket server. Then it’s an easy direct solution. Reference
encoding_errors=. It takes any of the following values - docs.python.org/3/library/codecs.html#error-handlersThe following works very well for me:
encoding = 'latin1'
latin1 eliminates all the errors because every possible byte maps to a valid character. Doesn't mean the characters are correct though.
latin is unique in being the only one without invalid bytes, the only one that can do bytes(range(256)).decode('latin1') without generating an error.The encoding for the file can be checked before passing the correct value for the encoding argument.
To get the encoding, a simple option in Windows is to open the file in Notepad++ and look at the encoding. The correct value for the encoding argument can then be found in the Python documentation.
Look at this question and the answers on Stack Overflow for more details on different possibilities to get the file encoding.
Use:
df = pd.read_csv("/content/data.csv", encoding='latin1')
Just add ,encoding='latin1' and it will work.
Using the code below works for me:
with open(keeniz_dir + '/world_cities.csv', 'r', encoding='latin1') as input:
latin1 eliminates all the errors because every possible byte maps to a valid character. Doesn't mean the characters are correct though.Don't pass an encoding option unless you are sure about the file encoding.
The default value encoding=None passes errors="replace" to the called open() function. Characters with encoding errors will be substituted with replacements, and you can then figure out the correct encoding or just use the resulting Dataframe.
If the wrong encoding is provided, pd will pass errors="strict" to open() and get a ValueError if the encoding is incorrect.
