InternalRedirect conflict with nested variables and decoding filter

Patch to identity test suite to demonstrate the problem

Patch to test suite demonstrating problem with InternalRedirect?

I've looked a little further into this, and it looks like the problem is caused by the use of InternalRedirect? (the IdentityFailure? exception is a subclass of cherrypy.InternalRedirect?). It is possible to trigger the problem without Identity by raising InternalRedirect? in a controller method which has had a parameter converted to a dict.

It looks like CherryPy? explicitly calls filters again after an internal redirect. In _cphttptools:

          # Loop to allow for InternalRedirect.
          while True:
              try:
                  applyFilters('before_main')
                  if self.execute_main:
                      self.main()
                  break
              except cherrypy.InternalRedirect, ir:
                  self.object_path = ir.path

I haven't checked, but I would have thought this would cause a problem with values that have been converted by validators as well, wouldn't it? For example, if you've converted a string to an integer, won't that cause a problem?

OK, further searching around has turned up ticket #1022, which references http://www.cherrypy.org/ticket/590. However, #1022 has been marked as closed/invalid, which I'm pretty sure is wrong. I think this is something that needs to be fixed in TurboGears. One way to fix it would be to ensure that after an InternalRedirect?, parameters are completely reparsed from the original request, instead of using values that have already been converted. This could cause a problem if people were causing side-effects during parameter parsing, but I think that would be pretty bad style anyway.

cherrypy.request.original_params is useful here. It can be passed as the 'params' argument to InternalRedirect? but it seems to be deprecated - I don't know whether it has an equivalent in CP3. Also, it triggers the #1340 problem, because it is already a unicode string, so it shouldn't be decoded...

Fixed in #3771. Thanks for the provided tests... Tests are always _greatly_ appreciated!

Correction: it is fixed in [3771]

ooops... or fixed in r3771 ? :)

The fix in changeset:3771 does not work for all possible (likely) nested variable structures. In particular in doesn't work where a dictionary is nested in a list which quite a common and desirable structure. Here is a solution which is more general.

         def encode_utf8(params):
            '''
            will recursively encode to utf-8 all values in a dictionnary
            what about when a dictionary is nested in a list?
            '''
            def encode_arg(value):
                if isinstance(value, dict):
                    for k,v in value.items():
                        value[k] = encode_arg(v)
                    return value
                elif isinstance(value, list):
                    return [encode_arg(vv) for vv in value]
                else:
                    return value.encode('utf-8')
            
            res = dict()
            for k, v in params.items():
                res[k] = encode_arg(v)
            return res

@salfield: This has been addressed in #1622 already.