[PATCH] Mako bytecode caching only works when use_dotted_templatenames = False

I assume you are actually referring to Mako's creation of .py (and .pyc) files from .mak files. If so, this is caused by not letting the Mako template loader actually load the file and reading the file and passing a string instead.

If you are referring to something else, let me know.

Also there is a bug in Mako that is related to this: http://www.makotemplates.org/trac/ticket/128 which I have also provided a patch for.

Fix mako caching problem

I just committed pushed this patch, which is similar to yours:

http://bitbucket.org/turbogears/tg-dev/changeset/020db0b5043e/

All of the test suite passes, and this doesn't require any Mako changes. Can you please test and verify that it still works for you in windows?

Thanks!

Ok, so my patch breaks custom templating.mako.compiled_templates_dir when used in conjunction with 'use_dotted_templatenames', since passing an absolute path to the mako Template(module_directory=) will still put them in myproject/templates/

Ok, I pushed out a couple of small tweaks, and everything seems to be working fine (aside from lambacck's windows-specific mako issue mentioned above). Closing...

Reopening, since I'm not 100% satisfied with how this works by default.

Right now out of the box (with this patch) use_dotted_templatenames is enabled and Mako stores your compiled bytecode in myproject/templates/the/full/path/to/myproject/templates/index.pyc. This behavior is documented by Mako (http://www.makotemplates.org/docs/usage.html#usage_file).

templating.mako.compiled_templates_dir defaults to your templates directory, and when use_dotted_templatenames is disabled Mako compiles and stores templates alongside the existing templates.

I don't know if we can easily have the same behavior with and without dotted templatenames, but I think we could do a better job of docmenting this and making it easy to change, without polluting the project's templates directory.

So, how about we add something like this in the default ini?

   {{if mako}}
   # The location to store your compiled templates
   templating.mako.compiled_templates_dir = %(here)/data/templates
   {{endif}}

This way, we're sharing %(here)/data with Beaker sessions and we're not touching the projects templates directory.

lmacken's comment applies to all Mako template caching and not just dotted names since we copied the behaviour of attempting to pick a suitable location from the non-dotted method.

I think that defaulting to explicit location is a good idea in general, because most users are going to want to have the compiled templates cached if only to assist with debugging. This means a better experience for new TG + Mako developers.

With the current code I don't think there is anyway to avoid writing the templates somewhere to disk (what happens when the process does not have write permissions?), so we might want to make no caching of templates an option at some point in the future. We might want to let "templating.mako.compiled_templates_dir = None" mean "don't cache the templates" or let the option be more explicit and don't cache the templates to disk unless a valid path is provided (maybe log a warning if there is a problem with the path).

In TG1, the equivalent of templating.mako.compiled_templates_dir defaulted to makotemplates (or something similar) in the development.ini but the line was commented out. If it was commented out then no template modules were written to disk.

Would you like me to submit patches for what is discussed above?

Replying to lambacck:

Would you like me to submit patches for what is discussed above?

Sure. Sorry, I meant to get this done before the weekend, but but caught up with other things. If you're willing and able to throw this patch together, please do so. If not, I can hack it together later this week. Cheers.

So I think this patch may be sufficient.

--- a/devtools/templates/turbogears/development.ini_tmpl
+++ b/devtools/templates/turbogears/development.ini_tmpl
@@ -68,18 +68,17 @@ templating.mako.reloadfromdisk = true
 
 # the compiled template dir is a directory that must be readable by your
 # webserver. It will be used to store the resulting templates once compiled
 # by the TemplateLookup system.
 # During development you generally don't need this option since paste's HTTP
 # server will have access to you development directories, but in production
 # you'll most certainly want to have apache or nginx to write in a directory
 # that does not contain any source code in any form for obvious security reasons.
-#
-#templating.mako.compiled_templates_dir = /some/dir/where/webserver/has/access
+templating.mako.compiled_templates_dir = %(here)s/data/templates
 
 # WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*
 # Debug mode will enable the interactive debugging tool, allowing ANYONE to
 # execute malicious code after an exception is raised.
 #set debug = false
 
 # Logging configuration
 # Add additional loggers, handlers, formatters here

The issue here is setting this value to None will create a 'None' directory to store your templates in. If we want None to default to in-memory template caching, we'll have to refactor some of the code in configuration.py which looks in a few places to see if it has the permissions to store templates there.

I pushed changeset 307 to set a sane default template cache directory.

If we want to handle None to only use an in-memory cache, this will still require a little bit more work.

actual tgdevtools-dev changeset 307

I threw together a quick patch that simplifies the heuristics of determining where to cache the compiled mako templates. If no cache directory is specified, it currently tries every pylons template path, and will then fall back to %(data)/templates. This patch removes these heuristics and makes the configuration more explicit. If templating.mako.compiled_templates_dir is not defined, None, False, or not writable, it will setup an in-memory cache for compiled template bytecode. Otherwise, it'll throw the cached templates wherever you specify.

I tested this with dotted template lookups, and filesystem based template lookups, both of which seem to work as expected. The only difference is that with dotted template lookups, the cache will have a different directory heirarchy than with path-based lookups. This shouldn't cause any problems as far as I can tell.

If someone else familiar with corner of TG could +1 this patch, that would be re-assuring.

--- a/tg/configuration.py       Sun Apr 18 16:26:13 2010 +0200
+++ b/tg/configuration.py       Sun May 16 15:51:41 2010 -0400
@@ -317,35 +317,26 @@
         # a file system based template lookup mechanism.
         compiled_dir = tg.config.get('templating.mako.compiled_templates_dir', None)
 
-        if not compiled_dir:
-            # Try each given templates path (when are they > 1 ?) for writability..
-            for template_path in self.paths['templates']:
-                if os.access(template_path, os.W_OK):
-                    compiled_dir = template_path
-                    break # first match is as good as any
-
-            # Last recourse: project-dir/data/templates (pylons' default directory)
-            if not compiled_dir:
+        if not compiled_dir or compiled_dir.lower() in ('none', 'false'):
+            # Cache compiled templates in-memory
+            compiled_dir = None
+        else:
+            bad_path = None
+            if os.path.exists(compiled_dir):
+                if not os.access(compiled_dir, os.W_OK):
+                    bad_path = compiled_dir
+                    compiled_dir = None
+            else:
                 try:
-                    root = os.path.dirname(os.path.abspath(self.package.__file__))
-                except AttributeError:
-                    # Thrown during unit tests when self.package.__file__ doesn't exist
-                    root = None
-
-                if root:
-                    pylons_default_path = os.path.join(root, '../data/templates')
-                    if os.access(pylons_default_path, os.W_OK):
-                        compiled_dir = pylons_default_path
-
-                if not compiled_dir:
-                    if use_dotted_templatenames:
-                        # Gracefully digress to in-memory template caching
-                        pass
-                    else:
-                        raise IOError("None of your templates directory, %s, are "
-                            "writable for compiled templates. Please set the "
-                            "templating.mako.compiled_templates_dir variable in your "
-                            ".ini file" % str(self.paths['templates']))
+                    os.makedirs(compiled_dir)
+                except:
+                    bad_path = compiled_dir
+                    compiled_dir = None
+            if bad_path:
+                log.warn("Unable to write cached templates to %r; falling back "
+                         "to an in-memory cache. Please set the `templating.mak"
+                         "o.compiled_templates_dir` configuration option to a "
+                         "writable directory." % bad_path)
 
         if use_dotted_templatenames:
             # Support dotted names by injecting a slightly different template

--- a/devtools/templates/turbogears/development.ini_tmpl        Sun May 16 12:51:38 2010 -0400
+++ b/devtools/templates/turbogears/development.ini_tmpl        Sun May 16 16:03:58 2010 -0400
@@ -66,13 +66,15 @@
 # deployed in an unzipped manner, then you can leave this option to true
 templating.mako.reloadfromdisk = true
 
-# the compiled template dir is a directory that must be readable by your
-# webserver. It will be used to store the resulting templates once compiled
-# by the TemplateLookup system.
+# the compiled template dir is a directory that must be readable and writable
+# by your webserver. It will be used to store the resulting templates once
+# compiled by the TemplateLookup system.
 # During development you generally don't need this option since paste's HTTP
 # server will have access to you development directories, but in production
 # you'll most certainly want to have apache or nginx to write in a directory
-# that does not contain any source code in any form for obvious security reasons.
+# that does not contain any source code in any form for obvious security
+# reasons.  If disabled, None, False, or not writable, it will fall back
+# to an in-memory cache.
 templating.mako.compiled_templates_dir = %(here)s/data/templates
 
 # WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*