Warning: Can't synchronize with repository "(default)" (Unsupported version control system "svn": No module named svn). Look in the Trac log for more information.

Ticket #430 (closed defect: fixed)

Opened 13 years ago

Last modified 12 years ago

[PATCH] The AutoCompleteField widget dosen't handle non-ascii text

Reported by: jens@… Owned by: anonymous
Priority: normal Milestone: 0.9a2
Component: TurboGears Version:
Severity: normal Keywords: jsonify json
Cc:

Description

When typing non-ascii characters in a AutoCompleteField, the search_controller will get the searchString as a ascii string, and not as a unicode string. This makes the returned list be shown wrong.

Simple workaround that works on my browser (FF 1.5 on Linux with sv_SE.UTF-8 locale) is to do: searchString = searchString.decode("utf-8") I'm not sure that this works in all cases, but something like this should work

I'm running svn-head as of today

Attachments

simplejson-dont-ensure-ascii.patch Download (399 bytes) - added by bjourne@… 13 years ago.

Change History

Changed 13 years ago by bjourne@…

comment:1 Changed 13 years ago by bjourne@…

I have the exact same problem, but the above patch fixes it. The problem seems to be caused by some encoding stuff in simplejson that doesn't coerce its output to unicode like the code for kid templates do. Adding ensure_ascii = False keeps the utf8 characters unescaped as they are when they are json-outputted. It makes it so they display correctly on my browser using utf8.

An alternative to this patch would be to cast the strings simplejson escapes to unicode.

comment:2 Changed 13 years ago by anonymous

  • Keywords jsonify json added
  • Component changed from Widgets to TurboGears

comment:3 Changed 13 years ago by roger.demetrescu

  • Summary changed from The AutoCompleteField widget dosen't handle non-ascii text to [PATCH] The AutoCompleteField widget dosen't handle non-ascii text

Just to make bjourne's patch appears in "{11} Pending Patches" report

comment:4 Changed 13 years ago by kevin

  • Status changed from new to closed
  • Resolution set to fixed

committed in [829]. Thanks!

comment:5 Changed 13 years ago by kevin

  • Status changed from closed to reopened
  • Resolution fixed deleted

Bob Ippolito correctly points out that this is the wrong fix:

Looks like it's a bug in TG somewhere.. simplejson does the right thing:

 >>> import simplejson
 >>> print simplejson.dumps(dict(text=u'\u00bfHabla espa\u00f1ol?'))
{"text":"\u00bfHabla espa\u00f1ol?"}

but TG does not:

 >>> import urllib
 >>> print urllib.urlopen('http://127.0.0.1:8080/test').read()
{"tg_flash":null, "text":"¿Habla español?"}

This is apparently Kevin's fault according to svn blame:

   370      kevin class GenericJSON(JSONEncoder):
   829      kevin     def __init__(self):
   829      kevin         super(GenericJSON, self).__init__
(ensure_ascii = False)

In this revision:
http://trac.turbogears.org/turbogears/changeset/829

Supposedly this "helps" non-ascii characters through JSON, but it
doesn't have the intended result.

The right fix for #430 would've been to ensure unicode strings from
browser input, instead of ensuring potential garbage makes its way
into the JSON output.  In this case, it would be a change to the way
URLs (at least the query string) are handled.

This could be fixed by making cherrypy.lib.parseQueryString look at
the resultant dict for keys that are outside of ASCII and decoding
them to UTF-8... or it could be done as a filter like the formencode
NestedVariablesFilter in startup.

Bob also points out the correct fix, which is closer to what's suggested in the original description of this ticket. The patch I applied is rolled back in [914].

comment:6 Changed 13 years ago by kevin

  • Status changed from reopened to closed
  • Resolution set to fixed
  • Milestone changed from 0.9 to 0.9a2

As of [993], the decodingFilter is turned on by default. This should take care of this problem.

Note: See TracTickets for help on using tickets.