Warning: Can't synchronize with repository "(default)" (Unsupported version control system "svn": No module named svn). Look in the Trac log for more information.

Ticket #1090 (closed defect: fixed)

Opened 13 years ago

Last modified 11 years ago

[PATCH] Safari XMLHttpRequest "fix" breaks JSON & other output

Reported by: FCodyC Owned by:
Priority: high Milestone: 1.5
Component: TurboGears Version: 0.9a5
Severity: normal Keywords: safari
Cc:

Description

I've done some testing with Safari and XMLHttpRequest, and here's what I've found.

1) Safari honors META tags' Content-Type attribute for character encoding. If you set it to "text/html; charset=utf8", (or your encoding of choice) non-ascii characters get displayed properly.

2) Safari honors HTTP Content-Type headers. If you set Content-Type here as above, Safari will render non-ascii characters properly.

3) For the URL of the current page, given no encodings in either HTTP headers or HTML META tags, Safari does some magic and still manages to display characters properly.

BUT: Here's where Safari seems to differ from Firefox and IE:

4) If a page is requested via XMLHttpRequest and it has no defined encoding in either HTTP or HTML, the charset is assumed latin1.

All that being said, the fix currently in place needs to be rewritten, because it actually introduces a rather annoying bug:

From trunk:

if ua.browser == "safari":

if isinstance(output, str):

output = output.decode(enc)

output = unicodechars.sub(

lambda m: "&#x%x;" % ord(m.group(1)), output).encode("ascii")

For all Content-Types that begin with "text/", this is encoding unicode characters as XML/HTML entities. That's not very appropriate for text/plain or text/x-json. I've seen many references to this "Safari bug". Why HTML entities would show up in JSON data is pretty confusing.

The fix is just to include the text-encoding in the Content-Type HTTP header if it's not currently present. (And to respect it if it IS present.)

Attachments

diff.txt Download (1.4 KB) - added by FCodyC 13 years ago.
Possible solution
diff2.txt Download (619 bytes) - added by joshua 12 years ago.
possible other solution

Change History

Changed 13 years ago by FCodyC

Possible solution

comment:1 Changed 13 years ago by FCodyC

I've added my possible change. I've got something similar working in 0.8.9, but I don't run trunk on my system so am unable to test it. I'll leave it to someone else to test. At the very least, the info about how Safari handles XMLHttpRequest will come in handy for the solution.

comment:2 Changed 13 years ago by jorge.vargas

  • Milestone set to 1.0b2

anyone with safari can verify this? or we close it.

comment:3 Changed 13 years ago by jorge.vargas

#1125 confirm this is still an issue, now is this patch the best way to fix it?

comment:4 Changed 13 years ago by kevin

The original Safari bug that that block of code was there to fix was that Safari was completely ignoring the encoding that it was being told about when doing XMLHTTPRequests. If that is no longer a bug in current (2.0+) Safaris, I'd be willing to just take out the block of code that as there. I don't have time to test that myself right now.

comment:5 Changed 12 years ago by alberto

  • Milestone changed from 1.0b2 to 1.1

Changed 12 years ago by joshua

possible other solution

comment:6 Changed 12 years ago by joshua

Works for me by just indenting two lines (see diff2.txt)

comment:7 Changed 12 years ago by alberto

  • Milestone changed from 1.1 to 1.0.1
  • Summary changed from Safari XMLHttpRequest "fix" breaks JSON & other output to [PATCH] Safari XMLHttpRequest "fix" breaks JSON & other output

Can anyone else please test this? (Or provide a sample quickstarted app so I can test it myself)

Alberto

comment:8 Changed 12 years ago by alberto

  • Status changed from new to assigned
  • Owner changed from anonymous to alberto

comment:9 Changed 12 years ago by alberto

  • Status changed from assigned to new
  • Owner alberto deleted
  • Summary changed from [PATCH] Safari XMLHttpRequest "fix" breaks JSON & other output to Safari XMLHttpRequest "fix" breaks JSON & other output

diff.txt doesn't apply cleanly, diff2.txt doesn't pass tests:

======================================================================
FAIL: test_safariUnicodeFix (turbogears.tests.test_controllers.TestRoot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alberto/src/turbogears/turbogears/tests/test_controllers.py", line 261, in test_safariUnicodeFix
    self.failUnlessEqual("¿Habla español?", firstline)
AssertionError: '¿Habla español?' != '\xc2\xbfHabla espa\xc3\xb1ol?'

An updated patch with a test-case would be appreciated. Thanks :)

Alberto

comment:10 Changed 12 years ago by alberto

  • Milestone changed from 1.0.1 to 1.0.2

comment:11 Changed 12 years ago by alberto

  • Milestone changed from 1.0.2 to 1.0.3

comment:12 follow-up: ↓ 13 Changed 12 years ago by MarkRamm

I believe the best approach to fix this is to delete the safari specific hack we have now (and the associated test).

The hack is there because the encoding was in content headers was being ignored by Safari 1.x, and that bug has been fixed in Safari 2. The hack is not compatible with the JSON code which returns a generator rather than a string. And it produces undesirable behavior (switching unicode strings in JSON to latin1).

So, unless someone objects I'll plan to remove the Safari encoding hack which is the source of the JSON problems people are seeing (along with the related test code) sometime later this week.

comment:13 in reply to: ↑ 12 Changed 12 years ago by FCodyC

Replying to MarkRamm:

So, unless someone objects I'll plan to remove the Safari encoding hack which is the source of the JSON problems people are seeing (along with the related test code) sometime later this week.

Yes, please do! I was disappointed to run into yet another bug regarding this chunk of code in the latest release. :(

comment:14 Changed 12 years ago by faide

  • Summary changed from Safari XMLHttpRequest "fix" breaks JSON & other output to [PATCH] Safari XMLHttpRequest "fix" breaks JSON & other output
  • Milestone changed from 1.0.3 to 1.1

comment:15 Changed 12 years ago by mramm

  • Status changed from new to closed
  • Resolution set to fixed

Duplicate of ticket 1284, fix applied in r3214 (1.0) and r3215 (1.1).

comment:16 Changed 11 years ago by poeml

  • Status changed from closed to reopened
  • Resolution fixed deleted

Hi,

it seems the fix breaks with newer Safaris. It definitely breaks my Safari, and I could assume that it is because I use a newer version. The version it reports is "Version 3.0.4 (523.12.2)", as it comes with 10.4.11 (the iLive 08 update might also be involved).

If I remove the fix, all works fine. (By removing, I mean commenting out the 5 lines in controllers.py with 'if ua.browser == "safari"'.)

if I don't remove it, I get the following (the code in use is "whatwhat"):

2008-02-12 14:12:51,968 turbogears.controllers DEBUG Calling <function remove_note at 0xb709c1b4> with *((<whatwhat.subcontrollers.project.ProjectController? object at 0xb70a0fcc>, u'10')), ({}) 2008-02-12 14:12:51,980 cherrypy.msg INFO HTTP: Page handler: <function _wrapper at 0xb58813ac> Traceback (most recent call last):

File "/usr/local/lib/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cphttptools.py", line 105, in _run

self.main()

File "/usr/local/lib/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cphttptools.py", line 254, in main

body = page_handler(*virtual_path, self.params)

File "/usr/local/lib/python2.4/site-packages/TurboGears-1.0.2.2-py2.4.egg/turbogears/identity/conditions.py", line 275, in _wrapper

return fn( *args, kw )

File "<string>", line 3, in remove_note File "/usr/local/lib/python2.4/site-packages/TurboGears-1.0.2.2-py2.4.egg/turbogears/controllers.py", line 334, in expose

output = database.run_with_transaction(

File "<string>", line 5, in run_with_transaction File "/usr/local/lib/python2.4/site-packages/TurboGears-1.0.2.2-py2.4.egg/turbogears/database.py", line 303, in so_rwt

retval = func(*args, kw)

File "<string>", line 5, in _expose File "/usr/local/lib/python2.4/site-packages/TurboGears-1.0.2.2-py2.4.egg/turbogears/controllers.py", line 351, in <lambda>

mapping, fragment, args, kw)))

File "/usr/local/lib/python2.4/site-packages/TurboGears-1.0.2.2-py2.4.egg/turbogears/controllers.py", line 391, in _execute_func

return _process_output(output, template, format, content_type, mapping, fragment)

File "/usr/local/lib/python2.4/site-packages/TurboGears-1.0.2.2-py2.4.egg/turbogears/controllers.py", line 100, in _process_output

output = unicodechars.sub(

TypeError?: expected string or buffer Request Headers:

REFERER:  http://myhost:8080/project/6 Content-Length: USER-AGENT: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit?/523.12.2 (KHTML, like Gecko) Version/3.0.4 Safari/523.12.2 CONNECTION: keep-alive COOKIE: active_project_id=6; show_closed_risks=0; tg-visit=83bbf717d8081548f067fd38a7dc073cd6cc24ab; utma=68994320.746445045.1187106567.1187106567.1187106567.1; utmz=68994320.1187106567.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); show_all_notes=1 HOST: myhost:8080 ACCEPT: */* Remote-Addr: 84.44.186.10 ACCEPT-LANGUAGE: en Content-Type: Remote-Host: 84.44.186.10 ACCEPT-ENCODING: gzip, deflate

I remember that I have seen this behaviour quite some time ago (a year maybe?). So it might not just happen with the very newest Safari... I just had no explanation for it at the time, but yesterday I looked into it and found the above.

comment:17 Changed 11 years ago by poeml

  • Status changed from reopened to closed
  • Resolution set to fixed

Sorry, I think I misinterpreted this issue completely, when I found this bug I didn't read it closely. It seems that I 1) didn't read this issue closely 2) am using an older version of TurboGears than I thought. It is 1.0.2.2 as installed in /usr/local/lib -- I thought I was using an up to date rpm on that machine.

So I'll close the issue again.

Sorry about the noise!

Note: See TracTickets for help on using tickets.