Warning: Can't synchronize with repository "(default)" (Unsupported version control system "svn": No module named svn). Look in the Trac log for more information.

Ticket #1168 (closed enhancement: fixed)

Opened 12 years ago

Last modified 11 years ago

[PATCH] Optimization of reloader_thread

Reported by: john.m.camara Owned by: anonymous
Priority: normal Milestone: 1.0b2
Component: TurboGears Version: 1.0b1
Severity: normal Keywords:
Cc:

Description

Changed a couple of lines a code to list comprehension to speed up reloader_thread function. With this patch the function runs nearly 100 times faster on my PC and reduces CPU load nearly 50 minutes per day.

Attachments

reloader_thread_optimized.patch Download (711 bytes) - added by john.m.camara 12 years ago.
reloader_thread_optimized.diff Download (744 bytes) - added by john.m.camara 12 years ago.
reloader_thread_optimized2.diff Download (995 bytes) - added by john.m.camara 12 years ago.

Change History

Changed 12 years ago by john.m.camara

comment:1 Changed 12 years ago by godoy

Can you make this patch with 'svn diff'? Or at least tell where's what you're patching?

The only "reloader*.py" I have here belongs to Paste.

Changed 12 years ago by john.m.camara

comment:2 Changed 12 years ago by john.m.camara

I ran svn diff for the original patch but ran the command from the wrong directory. Any way update patch corrects this.

comment:3 Changed 12 years ago by jorge.vargas

  • Priority changed from normal to high
  • Severity changed from normal to minor

the patch is agains this line http://trac.turbogears.org/turbogears/browser/tags/1.0b1/turbogears/startup.py#L48

and it's the result of this  http://tinyurl.com/yzyuux

if it doesn't what it says I'll like to have this in asap so I'll live it for b2

oh well you fix it already :)

comment:4 Changed 12 years ago by godoy

I didn't note all that difference in a real project. Actually, I can't say that there was any improvement... I'll time it with and without this patch and see.

comment:5 Changed 12 years ago by john.m.camara

  • Priority changed from high to normal
  • Severity changed from minor to normal

Are you running in dev or prod mode. I believe you will only see the improvement when running in dev mode.

comment:6 Changed 12 years ago by godoy

  • Priority changed from normal to high
  • Severity changed from normal to minor

(patched) ./start-siteamostras.py 6,59s user 1,16s system 82% cpu 9,405 total ./start-siteamostras.py 6,52s user 1,17s system 83% cpu 9,206 total ./start-siteamostras.py 6,58s user 1,14s system 64% cpu 11,874 total

(unpatched) ./start-siteamostras.py 6,81s user 1,15s system 79% cpu 10,029 total ./start-siteamostras.py 6,46s user 1,26s system 83% cpu 9,256 total ./start-siteamostras.py 6,53s user 1,19s system 71% cpu 10,875 total

comment:7 Changed 12 years ago by godoy

The system only restarts automatically in dev mode.

comment:8 Changed 12 years ago by john.m.camara

  • Priority changed from high to normal
  • Severity changed from minor to normal

reloader_thread_optimized2.diff is an improved patch

Changed 12 years ago by john.m.camara

comment:9 Changed 12 years ago by godoy

(patched - version2)
./start-siteamostras.py  6,56s user 1,14s system 80% cpu 9,546 total
./start-siteamostras.py  6,51s user 1,17s system 81% cpu 9,405 total

(unpatched)
./start-siteamostras.py  3,05s user 0,58s system 42% cpu 8,631 total
./start-siteamostras.py  6,53s user 1,17s system 84% cpu 9,110 total

comment:10 Changed 12 years ago by john.m.camara

using the hotshot profiler I get

unpatched - 0.363 secs patched v2 - 0.011 secs

Although when I use time.clock on a similar example outside of the TurboGears code I only see a 43% improvement rather than the 3300% improvement hotshot states.

comment:11 Changed 12 years ago by godoy

I believe that the problem is not the performance of this part alone but how it affects the whole project. I don't see any gain here... I've run each case at least 5 times, and taken the 2 or 3 last runs.

From what I've tested, I'd spend the time looking somewhere else.

If it stays like this, I'm "+/- 0" on this, i.e., if it gets applied it won't change anything, if it doesn't we won't be loosing anything... If there was some performance gain, I'd apply this myself, but I'm more on the conservative side here. :-)

comment:12 Changed 12 years ago by john.m.camara

It must be differences between our systems as I just ran an additional 30+ tests. About half on the whole project and the other half on just part of the project or similar examples but outside of TurboGears and in all cases I have had reasonable speedups. I even tried different versions of Python (2.4, 2.4.3, and 2.3.5).

Oh well, it would be interesting to see what results someone else gets.

BTW I have assumed that you do have autoreload.package set to your project name as if you had it set to None or if it was missing you would not see any performance gains. I figured that was a safe assumption to make.

comment:13 Changed 12 years ago by godoy

You should assume only what you get from quickstart. Anything else would be a custom configuration.

I'll check with other projects. It might be due to some characteristics of this particular project.

comment:14 Changed 11 years ago by alberto

  • Status changed from new to closed
  • Resolution set to fixed

Seems a better way to express the same thing and is more readable. If it really saves our CPU's 50 minutes a day (or even 5) I'm +1 for it.... Commited in [2257] Thanks :)

Alberto

Note: See TracTickets for help on using tickets.