4

I'm using BeautifulSoup, and I get back a string like this:

u'Dassault Myst\xe8re'

It's a unicode, but what I want is to make it look like:

'Dassault Mystère'

I have tried 

name = name.encode('utf-8'), decode(), unicode()

The error I keep getting is:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8'

My default encoding seems to be 'ascii': sys.getdefaultencoding() returns 'ascii' even though I have:

#!/usr/bin/env python
# encoding: utf-8

At the top of the file.

Hoping to solve this recurring Unicode issue once and for all!

Thanks

Improve this question

1 Answer 1

Reset to default
1

I do not know how and where you get this message, but look at this exmple:

$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> txt = u'Dassault Myst\xe8re'
>>> txt
u'Dassault Myst\xe8re'
>>> print txt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 13:
  ordinal not in range(128)
>>> ^D
$ export LANG=en_US.UTF-8
$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> txt = u'Dassault Myst\xe8re'
>>> txt
u'Dassault Myst\xe8re'
>>> print txt
Dassault Mystère
>>>^D

So as you can see if you have a console as ASCII then during print, there is a conversion from unicode to ascii, and if there is character outside ASCII scope - exception is thrown.

But if console can accept unicode, then everything is correctly displayed.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Well, that fixed the printing to console issue. But I still have a problem when building a url, because when I append - u'Dassault Myst\xe8re' to url urllib2 chokes on it when making a http request. I guess it's expecting an ascii string, and I'm sending something else?
My url looks like this: u'youtube.com/results?search_query=Dassault+Myst\xe8re&aq=0' and urllib2 doesn't like that it seems.
2nd part solved using this answer: stackoverflow.com/questions/4389572/…
I believe urllib2 works as expected with "from future import unicode_literals" .

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.