Archive for the ‘WSGI’ Category

Early impressions on modwsgi in production

July 21, 2009

At the end of May we made the KARL cutover from KARL2 to KARL3.  We have now had almost 8 weeks in production, so we can form impressions about some of the decisions.

For example, we’re using modwsgi as a WSGI server.  In a way, this was a surprising decision.  Chris and I were both a bit skeptical about whether it was a risk worth taking, as we hadn’t used it “in anger” beyond the site.  As somewhat a throwaway test, I set it up on the test site we used for OSI to do the user acceptance testing for the 15 or so milestone deliverables on KARL, but with the idea that it wasn’t a permanent decision.

By the time we setup the deployment server, we had months of living with modwsgi under our belt, somewhat by accident.  Far more important, we added Shane Hathaway to the project.  Like Chris Rossi, Shane has been a boon for KARL in many ways.  In this case, Shane had experience with modwsgi and said he would make sure we could stand behind it.

So we put it into production.  We’re running on an 8-core box setup as a Xen server, with the OSI instance getting 3 CPUs.  We started with modwsgi setup to run 2 BFG application processes (and thus ZEO clients), each with 2 threads.  modwsgi gave us some simplification: we didn’t need Apache/mod_proxy + BFG/Paster, with the latter managed by supervisor, possibly with a load balancer in between.  Instead, we let modwsgi in daemon mode handle that.

We also found that we could upgrade the software on the server, send a -GRACEFUL restart to Apache, and have a zero-downtime update to the server.  Which was nice.

Later, though, we found a really useful benefit to having modwsgi in the equation.  We then found that our LiveSearch implementation (and another part of the application) were slower (up to a second for a request) than other parts.  So we put in an Apache alias that matched on those URL types and sent them to a separately-configured BFG instance we call the “ghetto”.  This instance is primarily for catalog requests, so we changed the ZEO cache to be *very* high, but we also flush from the cache on each request any object that isn’t in the catalog.  This has been a boon.

I have an interest in looking later at some more options we gain from modwsgi.  For example, file delivery using wsgi.file_wrapper (which modwsgi fixed its implementation bugs in later releases.)  Also, modwsgi has some facilities for setting limits on the life of a request.

Still, regarding just the basics, we haven’t had much of a hitch at all.  So far, so good.


Belated happy first birthday, BFG

July 21, 2009

While I was on vacation, Chris McDonough wrapped up work on BFG 1.0.  The BFG elevator speech hits the nail on the head:

BFG is a “pay only for what you eat” Python web framework. You can get started easily and learn new concepts as you go, and only if you need them. It’s simple, well tested, well documented, and fast.

I’ve used BFG on KARL for the last 9 or so months and have found almost everything about it to be a relief. With this 1.0 release, BFG has achieved a sweet spot of stability and maturity combined with ongoing vitality. While BFG is particularly attractive to Zope developers looking for a modern alternative, it is also attractive more generally to Python web developers.  It’s hard to imagine there still being unmet needs, but BFG stands out for people who take the points in the elevator speech above seriously.

All of this due to the massive effort Chris put into it. Not just making it, but mega-documenting it, 100% coverage on tests, answering questions, sample applications, keeping everything up to date on every change, etc. There are quite a few people participating in BFG, but these efforts are generously given because Chris is there to integrate them.

So congrats Chris, and thanks.

Fun and profit with middleware

June 3, 2009

Yesterday we were working on the KARL project, doing some post-deployment housekeeping.  Specifically, we had a checkout of the templates that had a local customization (injecting Google Analytics) that we didn’t want to check in.  At least not to the software repository itself.  We wanted building a demo KARL to have no analytics, certainly not OSI’s account.

The most logical thing would have been to throw ZPT at it and get a little snippet of HTML to jam in just before closing the body tag.  But that would mean going to a number of places and injecting calls, plus we’d have to grab the configuration data from somewhere for the right snippet.

Tres and Chris Rossi argued for middleware: something that would watch the outgoing HTML and inject Google Analytics in the appropriate circumstances.  An hour later, Tres had written repoze.urchin that parameterized in the Paste configuration file the data, then hacked the HTML on the way out.

When to use and not use middleware is an art that I’m still learning about.  The biggest two rules appear to be, don’t solve a problem with middleware if the application won’t run without it, or if the middleware requires access to information inside the application.

Website for BFG: It’s Alive!

April 24, 2009

I’ve been working non-stop on a large BFG (*) app for a number of months.  Well, almost non-stop.  I spent a week with Chris on a different BFG project.

Thus, I didn’t do a dang thing to help it, but here it is: a shiny new website for the BFG microframework.  Congratulations to Carlos and Chris for getting this up and running.  (Carlos writes about the site launch.)

BFG has been pretty unique in a number of ways, one of which is apropos for this site launch: Chris actually published the copious, well-written BFG docs just before releasing BFG’s code.  He’s been insistent on keeping the docs up-to-date.

Having a site is a big help.  Thanks guys!

(*) BFG is a Python web application “micro” framework that takes patterns, ideas, and some code from WSGI, WebOb, and Zope.  Keywords: fast, light, documented, tested, and non-judgemental.