Warning: Can't synchronize with repository "(default)" (Unsupported version control system "svn": No module named svn). Look in the Trac log for more information.

Ticket #2133 (closed defect: fixed)

Opened 11 years ago

Last modified 9 years ago

AttributeError in beaker in tg2.0b2 python2.4

Reported by: lszyba1 Owned by: faide
Priority: normal Milestone: 2.0rc1
Component: TurboGears Version: trunk
Severity: normal Keywords:
Cc: toshio

Description (last modified by Chris Arndt) (diff)

on python2.4 with identity. Its either coded wrong or code for a wrong version?

Let me know. Right now tg2b2 does not work.

URL: http://localhost:8080/
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/WebError-0.9.1-py2.4.egg/weberror/evalexception.py',
line 428 in respond
 app_iter = self.application(environ, detect_start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/configuration.py',
line 521 in remover
 return app(environ, start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/repoze.tm2-1.0a3-py2.4.egg/repoze/tm/__init__.py',
line 19 in __call__
 result = self.application(environ, save_status_and_headers)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/repoze.who-1.0.9-py2.4.egg/repoze/who/middleware.py',
line 107 in __call__
 app_iter = app(environ, wrapper.wrap_start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/ToscaWidgets-0.9.5dev_20081026-py2.4.egg/tw/core/middleware.py',
line 36 in __call__
 return self.wsgi_app(environ, start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/ToscaWidgets-0.9.5dev_20081026-py2.4.egg/tw/core/middleware.py',
line 59 in wsgi_app
 resp = req.get_response(self.application)
File 'build/bdist.linux-i686/egg/webob/__init__.py', line 1306 in get_response
File 'build/bdist.linux-i686/egg/webob/__init__.py', line 1275 in
call_application
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/ToscaWidgets-0.9.5dev_20081026-py2.4.egg/tw/core/resource_injector.py',
line 67 in _injector
 resp = req.get_response(app)
File 'build/bdist.linux-i686/egg/webob/__init__.py', line 1306 in get_response
File 'build/bdist.linux-i686/egg/webob/__init__.py', line 1275 in
call_application
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/middleware.py',
line 81 in __call__
 return self.app(environ, start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/middleware.py',
line 159 in __call__
 return self.wrap_app(environ, session_start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Routes-1.10.1-py2.4.egg/routes/middleware.py',
line 118 in __call__
 response = self.app(environ, start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Pylons-0.9.7rc4-py2.4.egg/pylons/wsgiapp.py',
line 117 in __call__
 response = self.dispatch(controller, environ, start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Pylons-0.9.7rc4-py2.4.egg/pylons/wsgiapp.py',
line 316 in dispatch
 return controller(environ, start_response)
File '/home/lucas/tmp/maa3/maa3/lib/base.py', line 36 in __call__
 return TGController.__call__(self, environ, start_response)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Pylons-0.9.7rc4-py2.4.egg/pylons/controllers/core.py',
line 204 in __call__
 response = self._dispatch_call()
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Pylons-0.9.7rc4-py2.4.egg/pylons/controllers/core.py',
line 159 in _dispatch_call
 response = self._inspect_call(func)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Pylons-0.9.7rc4-py2.4.egg/pylons/controllers/core.py',
line 95 in _inspect_call
 result = self._perform_call(func, args)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/controllers.py',
line 507 in _perform_call
 result = DecoratedController._perform_call(
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/controllers.py',
line 118 in _perform_call
 response = self._render_response(controller, output)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/controllers.py',
line 281 in _render_response
 template_name=template_name)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/render.py',
line 102 in render
 template_vars.update(_get_tg_vars())
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/render.py',
line 53 in _get_tg_vars
 flash = tg.get_flash(),
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/TurboGears2-2.0b2-py2.4.egg/tg/flash.py',
line 27 in get_flash
 msg = session.get('flash_message', '')
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Paste-1.7.2-py2.4.egg/paste/registry.py',
line 137 in __getattr__
 return getattr(self._current_obj(), attr)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/session.py',
line 463 in __getattr__
 return getattr(self._session(), attr)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/session.py',
line 458 in _session
 self.__dict__['_sess'] = Session(req, use_cookies=True,
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/session.py',
line 86 in __init__
 self.cookie = SignedCookie(secret, input=cookieheader)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/session.py',
line 36 in __init__
 Cookie.BaseCookie.__init__(self, input)
File 'Cookie.py', line 568 in __init__
 if input: self.load(input)
File 'Cookie.py', line 621 in load
 self.__ParseString(rawdata)
File 'Cookie.py', line 651 in __ParseString
 rval, cval = self.value_decode(V)
File '/home/lucas/tmp/tg2b2env/lib/python2.4/site-packages/Beaker-1.1.2-py2.4.egg/beaker/session.py',
line 40 in value_decode
 sig = hmac.new(self.secret, val[40:], sha1).hexdigest()
File 'hmac.py', line 107 in new
 return HMAC(key, msg, digestmod)
File 'hmac.py', line 42 in __init__
 self.outer = digestmod.new()
AttributeError: 'builtin_function_or_method' object has no attribute 'new'

Attachments

beaker-hmac2.4.patch Download (2.8 KB) - added by toshio 10 years ago.
version checking to ensure compatibility of sha and hmac imports

Change History

comment:1 Changed 11 years ago by Chris Arndt

  • Description modified (diff)

Hi Lukasz, please try to format your tickets with nice Trac wiki syntax...

comment:2 follow-up: ↓ 7 Changed 11 years ago by pjenvey

What is beaker.session.sha1 in this environment? It should be the sha module on Python 2.4, which should work with hmac.new. The sha module should have a new function

Without knowing more this traceback suggests your environment may be the culprit

comment:3 Changed 11 years ago by mramm

  • Milestone changed from 2.0b2 to 2.0b3

comment:4 follow-up: ↓ 5 Changed 10 years ago by mramm

  • Status changed from new to closed
  • Resolution set to worksforme

Can anybody reproduce this? I tried with an old python, and wasn't able to break it.

--Mark

comment:5 in reply to: ↑ 4 Changed 10 years ago by mvalente

  • Priority changed from highest to high
  • Status changed from closed to reopened
  • Resolution worksforme deleted

Replying to mramm:

Can anybody reproduce this? I tried with an old python, and wasn't able to break it.

--Mark

The same error happens to me. On python 2.4.4 and TG2.0b5.1

Showed up when I was following the Wiki tutorial, at the "save" method

-- MV

comment:6 Changed 10 years ago by pjenvey

mvalente -

Could you please provide the information I asked for above? Also, with what version of Beaker?

comment:7 in reply to: ↑ 2 Changed 10 years ago by mvalente

Replying to pjenvey:

What is beaker.session.sha1 in this environment? It should be the sha module on Python 2.4, which should work with hmac.new. The sha module should have a new function

In my case:

Module: beaker.session file: /usr/local/tg2/lib/python2.4/site-packages/Beaker-1.1.3-py2.4.egg/beaker/session.py

These are the last lines of the traceback:

File '/usr/local/tg2/lib/python2.4/site-packages/Beaker-1.1.3-py2.4.egg/beaker/session.py', line 481 in repr

return self._session().repr()

File '/usr/local/tg2/lib/python2.4/site-packages/Beaker-1.1.3-py2.4.egg/beaker/session.py', line 458 in _session

self.dict_sess? = Session(req, use_cookies=True,

File '/usr/local/tg2/lib/python2.4/site-packages/Beaker-1.1.3-py2.4.egg/beaker/session.py', line 86 in init

self.cookie = SignedCookie?(secret, input=cookieheader)

File '/usr/local/tg2/lib/python2.4/site-packages/Beaker-1.1.3-py2.4.egg/beaker/session.py', line 36 in init

Cookie.BaseCookie?.init(self, input)

File '/usr/lib64/python2.4/Cookie.py', line 568 in init

if input: self.load(input)

File '/usr/lib64/python2.4/Cookie.py', line 621 in load

self.ParseString(rawdata)

File '/usr/lib64/python2.4/Cookie.py', line 651 in ParseString

rval, cval = self.value_decode(V)

File '/usr/local/tg2/lib/python2.4/site-packages/Beaker-1.1.3-py2.4.egg/beaker/session.py', line 40 in value_decode

sig = hmac.new(self.secret, val[40:], sha1).hexdigest()

File '/usr/lib64/python2.4/hmac.py', line 107 in new

return HMAC(key, msg, digestmod)

File '/usr/lib64/python2.4/hmac.py', line 42 in init

self.outer = digestmod.new()

AttributeError?: 'builtin_function_or_method' object has no attribute 'new'

-- MV

comment:8 follow-up: ↓ 9 Changed 10 years ago by pjenvey

I'd also like to know what beaker.session.sha1 is in this environment, i.e. show me the output of something like:

Python 2.4.5 (#1, Apr 25 2008, 15:16:02) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.get_distribution('Beaker').version
'1.1.3'
>>> import beaker.session
>>> beaker.session.sha1
<module 'sha' from '/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-dynload/sha.so'>
>>> import hmac
>>> hmac.new('test', 'test', beaker.session.sha1).hexdigest()
'0c94515c15e5095b8a87a50ba0df3bf38ed05fe6'
>>> 

comment:9 in reply to: ↑ 8 Changed 10 years ago by mvalente

Replying to pjenvey:

I'd also like to know what beaker.session.sha1 is in this environment, i.e. show me the output of something like:

(tg2)[root@localhost wiki20]# python Python 2.4.4 (#1, Oct 23 2006, 13:58:18) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import pkg_resources pkg_resources.get_distribution('Beaker').version

'1.1.3'

import beaker.session beaker.session.sha1

<built-in function new>

import hmac hmac.new('test', 'test',beaker.session.sha1).hexdigest()

Traceback (most recent call last):

File "<stdin>", line 1, in ? File "/usr/lib64/python2.4/hmac.py", line 107, in new

return HMAC(key, msg, digestmod)

File "/usr/lib64/python2.4/hmac.py", line 42, in init

self.outer = digestmod.new()

AttributeError?: 'builtin_function_or_method' object has no attribute 'new'

comment:10 Changed 10 years ago by lszyba1

same results..I've upgraded to a newer version to see if that has changed.

>>> import pkg_resources
>>> pkg_resources.get_distribution('Beaker').version
'1.2.1'
>>> import beaker.session
>>> beaker.session.sha1
<built-in function openssl_sha1>
>>> import hmac
>>> hmac.new('test','test','beaker.session.sha1').hexdigest()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.4/hmac.py", line 107, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib/python2.4/hmac.py", line 42, in __init__
    self.outer = digestmod.new()
AttributeError: 'str' object has no attribute 'new'

comment:11 Changed 10 years ago by lszyba1

I've just noticed that the error is in hmac.py in my global python:

/usr/lib/python2.4/hmac.py

What is hmac.py? I wonder if it changed from 2.4 to 2.5?

Lucas

comment:12 Changed 10 years ago by pjenvey

lszyba1 your attempt was incorrect, you mistakenly passed the string 'beaker.session.sha1' to hmac instead of the actual variable beaker.session.sha1

This output tells me that Beaker was able to import the hashlib module -- and that beaker.session.sha1 is hashlib.sha1. However hashlib wasn't present in Python until version 2.5 (and the hmac module wasn't changed to handle the new hashlib APIs until 2.5)

So is there indeed a hashlib module present on your environments? If so, why, if you're using Python 2.4? Do you have the hashlib module from pypi installed?  http://pypi.python.org/pypi/hashlib

Maybe Beaker needs to hardcode the Python version number instead of attempting checking for the availability of hashlib (which would suck, because various other Pylons/TG related dependencies would need to be changed to do this as well)

comment:13 Changed 10 years ago by lszyba1

hashlib module is installed separately. easy_install hashlib for python2.4.

So what you are saying is that in code the hmac module is calling hashlib function. But the way it calls it is only supported on python2.4?

Can the code be changed via an if statement and use 2.4 api instead? Where would that be in a code then?

Lucas

comment:14 Changed 10 years ago by lszyba1

It calls python2.5 api instead of checking the version and use different api for python2.4? I meant...

comment:15 Changed 10 years ago by jorge.vargas

  • Priority changed from high to normal
  • Milestone changed from 2.0b5 to 2.0

moving this to 2.0 as it is really a change in external api (beaker) therefore ones it's closed it means we'll need to bump the version of beaker, actually make pylons bump it.

comment:16 Changed 10 years ago by lszyba1

I've emailed beaker maintainers. I have a app ready to be deployed on py2.4 and I can't do it because of this issue.

Thanks, Lucas

comment:17 follow-up: ↓ 24 Changed 10 years ago by bbangert

This shouldn't be fixed in Python. The maintainer of the hashlib package that you installed should really fix this. It isn't a beaker compatibility issue, its due to the hashlib module you installed (which is why our Python 2.4 buildbot did not see this issue).

Note:  http://www.google.com/codesearch?as_q=hmac+hashlib+sha1+ImportError&btnG=Search+Code&hl=en&as_lang=&as_license_restrict=i&as_license=&as_package=&as_filename=\.py%24&as_case=

There is a lot of packages that look for hashlib like we do, NONE of them will work on your system as long as you have that hashlib package that doesn't work like the Python 2.5 and later one.

Are you really suggesting that everyone put in a decent amount of try/except's along with other various feature tests to deal with *one* guy's hashlib package that uses the same namespace as a later Python versions and thus causes this whole problem?

I mean, its possible that a proper solution might be that everyone goes and does this.... but I'd like to believe this is a situation where the hashlib package for Python 2.4 users should monkey-patch hmac so that its used properly, as feature testing around the issue is rather ugly.

comment:18 Changed 10 years ago by lszyba1

Thanks for the explanation. I am using a python2.4 shipped with debian. I guess at this point because debian released a new version of its stable distribution, people like me should upgrade. For anybody that have sarge installed and want to run tg2 this will be a problem.

It would be nice if I could overwrite that file. Can I copy that file from py2.5 and use it in 2.4 without issues? I assume so since py2.4 doesn't have it by default.

Thanks, Lucas

comment:19 Changed 10 years ago by bbangert

I don't believe so, for the same reason that the hashlib package isn't working right. The way the Python 2.4 hmac module calls it apparently isn't the same way as the Python 2.5 hmac module calls it (it being the hashlib sha function).

I'd suggest either not using it, or using it under a different name so that these packages don't think they have a hashlib that works with hmac.

comment:20 Changed 10 years ago by lszyba1

how would you suggest I do that?

right now I install hashlib via easy_install hashlib

At this point I'm looking for a solution that will work, since we know what the problem is.

comment:21 Changed 10 years ago by pjenvey

The 2.5 hmac doesn't have many changes from 2.4s (it's really just tweaked to support hashlib). I'd try monkey patching it, you might get lucky. Bring the 2.5 hmac module into your app, and then as early as possible in your app's initialization do:

import hmac
from myapp.lib.hmac25 import HMAC as HMAC25
hmac.HMAC = HMAC25

If this works you may want to suggest to the author of the Pypi hashlib that his hashlib do this monkey patching for you, or at least suggest it in his documentation

comment:22 Changed 10 years ago by aigarius

Copying hmac.py file from python2.5 distribution to tg2env/lib/python2.4/ works for me.

comment:23 Changed 10 years ago by mramm

  • Status changed from reopened to closed
  • Resolution set to wontfix

comment:24 in reply to: ↑ 17 ; follow-up: ↓ 29 Changed 10 years ago by toshio

Replying to bbangert:

There is a lot of packages that look for hashlib like we do, NONE of them will work on your system as long as you have that hashlib package that doesn't work like the Python 2.5 and later one.

Note that this is incorrect on two counts. 1) the hashlib package is a drop in replacement for the python-2.5 and later one. That's not the problem here. 2) MOST modules that check for hashlib like this will work fine. The problem only arises when code that checks for hashlib like this then sends the created object into code that can only work with the old md5 and sha objects, not the new objects from hashlib.

In third-party modules, you could ask that those modules be changed to accept the new-style hashlib objects. Since hmac comes in the python2.4 stdlib it's not realistic to think that this will be changed there. Especially because the API documents its expectations and Beaker is breaking it. If Beaker wants to be compatible with python-2.4, it needs to only pass in the old-style md5/hash objects if it's using the 2.4 hmac. This means you have to resort to an ugly python version check where you're importing md5/sha/hashlib to determine what hmac version you're working with: (if sys.version_info[0:2] == (2,4): # old style else: # new style) but that's a limitation of hmac.py not providing a separate version.

This check belongs in Beaker since it's Beaker that's deciding what object to instantiate and then pass on to hmac. The current Beaker logic is choosing which version of hmac it is coded for based on the availability of hashlib. This is flawed logic - instead Beaker needs to check what hmac is installed (via the python version check) to make the decision.

Are you really suggesting that everyone put in a decent amount of try/except's along with other various feature tests to deal with *one* guy's hashlib package that uses the same namespace as a later Python versions and thus causes this whole problem?

I mean, its possible that a proper solution might be that everyone goes and does this.... but I'd like to believe this is a situation where the hashlib package for Python 2.4 users should monkey-patch hmac so that its used properly, as feature testing around the issue is rather ugly.

While that's another solution, having a third-party library that monkey patches stdlib libraries seems like a bad design decision. When an application chooses to do this at least the coder knows that they've changed the semantics of something basic to the python distribution they're using. But for a library to do this can lead to all sorts of problems: user expectations, unintentionally breaking unrelated code that depended on that version, etc.

I'm not going to reopen this ticket as in Fedora we only support python-2.4 in EPEL, not Fedora proper and we'll modify the Beaker packages locally to support the old hmac API there. Just wanted to point out for people googling this that this is not hashlib's problem. It's a problem with passing the incorrect type to a specific hmac API version because the Beaker's detection logic is flawed.

comment:25 Changed 10 years ago by pjenvey

If you want to patch around Beaker, go ahead. We'll even accept an upstream patch to work around this

But note that Beaker is only one of the few libraries that triggers the problem. You're going to have to patch all the others too. You can spot some of them via google code search:

 http://www.google.com/codesearch?hl=en&lr=&q=from+import+hashlib+hmac%5C.new&sbtn=Search

I don't like monkey patching either, but IMHO it's the the most practical solution. Especially when it only needs to happen on systems with the 2.4 hashlib installed, of which there are few.

comment:26 Changed 10 years ago by toshio

  • Cc toshio added
  • Status changed from closed to reopened
  • Resolution wontfix deleted

Sorry, I thought I cc'd myself but I was wrong :-(. Attaching a patch to beaker-1.3.1 that does the version check. You're right about changes being needed in other people's code as well. Any module that claims compatibility with python-2.4, uses hmac and hashlib with a fallback to sha/md5, and doesn't check the version is susceptible to this problem. Since TG2 requires hashlib on 2.4 as of r6091 this affects more people than I thought.

Changed 10 years ago by toshio

version checking to ensure compatibility of sha and hmac imports

comment:27 Changed 10 years ago by bbangert

  • Status changed from reopened to closed
  • Resolution set to fixed

I've applied this patch to Beaker tip. Beaker 1.4 being released tomorrow with this fix.

comment:28 Changed 10 years ago by bbangert

Thanks for the patch as well. :)

comment:29 in reply to: ↑ 24 Changed 9 years ago by mike3050

Replying to toshio:

Replying to bbangert:

There is a lot of packages that look for hashlib like we do, NONE of them will work on your system as long as you have that hashlib package that doesn't work like the Python 2.5 and later one.

Note that this is incorrect on  home insurance quotes two counts. 1) the hashlib package is a drop in replacement for the python-2.5 and later one. That's not the problem here. 2) MOST modules that check for hashlib like this will work fine. The problem only arises when code that checks for hashlib like this then sends the created object into code that can only work with the old md5 and sha objects, not the new objects from hashlib.

In third-party modules, you could ask that those modules be changed to accept the new-style hashlib objects. Since hmac comes in the python2.4 stdlib it's not realistic to think that this will be changed there. Especially because the API documents its expectations and Beaker is breaking it. If Beaker wants to be compatible with python-2.4, it needs to only pass in the old-style md5/hash objects if it's using the 2.4 hmac. This means you have to resort to an ugly python version check where you're importing md5/sha/hashlib to determine what hmac version you're working with: (if sys.version_info[0:2] == (2,4): # old style else: # new style) but that's a limitation of hmac.py not providing a separate version.

This check belongs in Beaker since it's Beaker that's deciding what object to instantiate and then pass on to hmac. The current Beaker logic is choosing which version of hmac it is coded for based on the availability of hashlib. This is flawed logic - instead Beaker needs to check what hmac is installed (via the python version check) to make the decision.

Are you really suggesting that everyone put in a decent amount of try/except's along with other various feature tests to deal with *one* guy's hashlib package that uses the same namespace as a later Python versions and thus causes this whole problem?

I mean, its possible that a proper solution might be that everyone goes and does this.... but I'd like to believe this is a situation where the hashlib package for Python 2.4 users should monkey-patch hmac so that its used properly, as feature testing around the issue is rather ugly.

While that's another solution, having a third-party library that monkey patches stdlib libraries seems like a bad design decision. When an application chooses to do this at least the coder knows that they've changed the semantics of something basic to the python distribution they're using. But for a library to do this can lead to all sorts of problems: user expectations, unintentionally breaking unrelated code that depended on that version, etc.

I'm not going to reopen this ticket as in Fedora we only support python-2.4 in EPEL, not Fedora proper and we'll modify the Beaker packages locally to support the old hmac API there. Just wanted to point out for people googling this that this is not hashlib's problem. It's a problem with passing the incorrect type to a specific hmac API version because the Beaker's detection logic is flawed.

Thank you.

Note: See TracTickets for help on using tickets.