This post has the jaded volume turned to eleven but even so, I’m impressed.

I watched the video of Rob Gietema’s presentation on Deco, the “new way to manage page layout, composite pages and rich content in Plone 4″.  I approach the entire topic with cynicism, dread, and exhaustion on a number of fronts.

Well, color me intrigued.  I’ll probably clutch tightly to my bitter, curmudgeonly outlook.  But I must confess to being impressed, in a number of ways.  I think they’ve done a good job thinking about the problem.  It looks like they’re going slow and being unafraid to refactor ideas and implementations.  It also has, already, a nice visual appeal along with some clever simplifying assumptions to keep the congitive overload under control.  Finally, this is a hard space to work in, technically, and it seems like they have some mad skillz.

There’s still a lot of challenge ahead.  The biggest is what I view as the chief paradox confronting Plone.  On one hand, a significant portion of people are fed up with how some of the features are implemented.  But on the other hand, they’re burned out by a legacy of undead overhauls and don’t have much patience for revolution.  (Sidebar: my opinion is, I sympathize but The Time Has Come.)

Additionally, while it’s a lot of work to do the product, it’s a lot more work to do the “whole” product: documentation, bug fixing, ongoing compatibility, performance, and other stuff over the long haul.  Letting Deco have a long gestation period outside the core would be advisable.  The more baked it is, the more legitimate it will feel when added to Plone, and the less resistance from the undead-overhaul-worriers.

So good luck Deco.  If even 2/3 of the features are implemented, but implemented superbly over the long haul, then Plone will have something sweet to hang its hat on.  (All of the above applies equally to Dexterity, the new content type system.)

Per the announcement, congrats to the new board of directors for the Plone Foundation.  I really, really like that lineup of people. It’s amazing, though, that there were at least five others in the election that would also have been strong to have on the board.

A particular thanks to Hanno Schlichting for once again conducting the election.  I have utter confidence in the things he does, be it Plone Foundation process stuff or gradually untangling years of technical cruft as part of his Plone 5 release manager duties.

I saw the minutes of the Plone Foundation annual member’s meeting (thanks Maurits for the writeup.)  Toby reported that, after spending half of the original CA donation of $100k, the PF in one year has its bank account back to the original amount.  This is primarily due to the sponsorship plan that Jon Stahl helped kick off, tied to the excellent work the Plone.net team at Pilot Sytems (plus Reinout, plus others) have done over the years.

The price point for sponsorship makes it easy to get companies in, and the allocation of funds (e.g. release managers, promotion) makes it very easy to justify.  Since the price point is a yearly fee, I suspect that you’ll see another $50k come in during the next half year, meaning…

…the Plone Foundation is a very successful community-managed outfit.  Congrats to the previous board for orchestrating this turnaround on finances, and here’s to a good 2010 for the Plone Foundation.

Dear Lazyweb,

Marius (and others) reported that Planet Python screws up the links on my articles.  Instead of a full link to the article, it instead does a href="".  Looking at my RSS feed, I see that it correctly has link>http://pauleveritt.wordpress.com/2009/07/15/kudos-to-malthe-for-chameleon/ in the item entries. My feed URL passes the validator test as well.

As you well know, Lazyweb, I switched to WordPress (hosted!) so I wouldn’t have to think, as I’m not very good at that particular trait.  Any ideas on the cause of this?

At the end of May we made the KARL cutover from KARL2 to KARL3.  We have now had almost 8 weeks in production, so we can form impressions about some of the decisions.

For example, we’re using modwsgi as a WSGI server.  In a way, this was a surprising decision.  Chris and I were both a bit skeptical about whether it was a risk worth taking, as we hadn’t used it “in anger” beyond the repoze.org site.  As somewhat a throwaway test, I set it up on the test site we used for OSI to do the user acceptance testing for the 15 or so milestone deliverables on KARL, but with the idea that it wasn’t a permanent decision.

By the time we setup the deployment server, we had months of living with modwsgi under our belt, somewhat by accident.  Far more important, we added Shane Hathaway to the project.  Like Chris Rossi, Shane has been a boon for KARL in many ways.  In this case, Shane had experience with modwsgi and said he would make sure we could stand behind it.

So we put it into production.  We’re running on an 8-core box setup as a Xen server, with the OSI instance getting 3 CPUs.  We started with modwsgi setup to run 2 BFG application processes (and thus ZEO clients), each with 2 threads.  modwsgi gave us some simplification: we didn’t need Apache/mod_proxy + BFG/Paster, with the latter managed by supervisor, possibly with a load balancer in between.  Instead, we let modwsgi in daemon mode handle that.

We also found that we could upgrade the software on the server, send a -GRACEFUL restart to Apache, and have a zero-downtime update to the server.  Which was nice.

Later, though, we found a really useful benefit to having modwsgi in the equation.  We then found that our LiveSearch implementation (and another part of the application) were slower (up to a second for a request) than other parts.  So we put in an Apache alias that matched on those URL types and sent them to a separately-configured BFG instance we call the “ghetto”.  This instance is primarily for catalog requests, so we changed the ZEO cache to be *very* high, but we also flush from the cache on each request any object that isn’t in the catalog.  This has been a boon.

I have an interest in looking later at some more options we gain from modwsgi.  For example, file delivery using wsgi.file_wrapper (which modwsgi fixed its implementation bugs in later releases.)  Also, modwsgi has some facilities for setting limits on the life of a request.

Still, regarding just the basics, we haven’t had much of a hitch at all.  So far, so good.

While I was on vacation, Chris McDonough wrapped up work on BFG 1.0.  The BFG elevator speech hits the nail on the head:

I’ve used BFG on KARL for the last 9 or so months and have found almost everything about it to be a relief. With this 1.0 release, BFG has achieved a sweet spot of stability and maturity combined with ongoing vitality. While BFG is particularly attractive to Zope developers looking for a modern alternative, it is also attractive more generally to Python web developers.  It’s hard to imagine there still being unmet needs, but BFG stands out for people who take the points in the elevator speech above seriously.

All of this due to the massive effort Chris put into it. Not just making it, but mega-documenting it, 100% coverage on tests, answering questions, sample applications, keeping everything up to date on every change, etc. There are quite a few people participating in BFG, but these efforts are generously given because Chris is there to integrate them.

So congrats Chris, and thanks.

A blog post that I’ve thought about many times and, shamefully, never quite got around to posting.

For the last 18+ months, Malthe Borch has been working on templating for Python.  Not the traditional mode of blank-slate-write-my-own, boy-this-is-fun.  But instead, superfast implementations of existing templating languages (ZPT, Genshi) with lots of tests and rapid bug fixing.

First with z3c.pt and then Chameleon (core and zpt), the amount of work he has done is staggering.  Look at the changelogs on those PyPI pages: 26 releases for z3c.pt, 41 for chameleon.core, 19 for chameleon.zpt, and 18 for sourcecodegen.  Big-time numbers.

If you’ve followed what he’s done, it’s even more than that.  He’s done quite a number of “well, that’s not working out, let’s discuss a re-implementation” (e.g. ditching lxml as a troublesome dependency via libxml2) efforts.  He spent a ton of time working with Sidnei da Silva on unit tests and test breakage for ZPT conformance.  He participates heavily in discussions not just about Chameleon, but also projects that interact with it or depend on it (Repoze, Plone, Zope.)

Our KARL project has been based on Chameleon from the beginning, ever since Chris McDonough went down the rabbit hole one weekend, off the clock, experimenting with a port to then-nascent BFG. Basing an application rewrite on a new template implementation would normally be a red flag.  But with Malthe’s attention to detail (plus Chris working on Chameleon too), everything was very smooth.  We got the huge benefit without much downside.

We as a community have some core software that we all use, over and over, without much thought to its provenance.  A lot of it comes from Jim Fulton (ZODB, buildout, etc.)  Chameleon has become like that.  It’s almost like the electricity: we all use it, it’s super-reliable, and we’ll never appreciate it enough until something goes wrong.

Here’s to some rightful appreciation for Malthe’s work.

In late May I made a brief blog post announcing KARL prior to the Plone Symposium talk I gave.  Since then, quite a bit has gone on with KARL:

• The migration went super-smooth.  We did 5 trial runs of gradually larger audiences, fixing bugs in between.  By the time of launch, Chris Rossi and Lars Nolan had the process down pat.  No surprises.
• We did a KARL 3.1 development cycle in support of hosting KARL for 2 more organizations.  In this process we did a good amount of refactoring, including parameterizing some customization points that previously required overriding via ZCML.
• Shane added an in-the-core People Directory.
• Fixed bugs that got overlooked.  This also went very smooth: most bugs got fixed within a day or two, and none piled up for more than a week, even the small ones.
• During the next two weeks, we are migrating the other two organizations from their existing KARL2 over to their new KARL3.  All hosted in the same environment with OSI, which makes operations far better.
• We’re also in the 3.2 development cycle.  We updated TinyMCE and landed profile picture resizing (via PIL).  We’re doing spellchecking, improvements to the email alerting, and a good number of other enhancements during the next few weeks.
• There are also plans for a big KARL meeting at OSI later this month, to include some other organizations.

All in all, we’ve really hit our stride.  Not just on development, but increasingly on operations.  We’re getting into a mode that is very measured, methodical, quality, responsive, etc.  As Jeffrey Shell always said: no alarms, no surprises.

Warning: blog abuse.  Just a permanent reminder to self.

A while back I asked Tres how to do a ternary-like operation in ZPT.  I needed to assign a value for a CSS class, where the value was one string in one case and another string for all other cases.

Tres gave me:

<div
class="blogEntry ${repeat['entry'].start and 'noborder' or ''}"
tal:repeat="entry entries">

(Updated July 15) Malthe wrote in a comment that he just changed Chameleon to allow the ternary syntax of ${'foo' if bar else 'boo'} ). Look in the sourcecodegen 0.6.11 release.)

Got back from 9 days with my wife’s family in France.  Wonderful vacation, especially for the kids (who went before us and stayed after us.)  We both got a head cold just before returning, so the two flights and aftermath were south of pleasant.

While out, longtime-friend Seb Bacon pointed out that the installation docs for KARL had bitrotted, a point which Chris had to cover for me on the latest repozecast.  Now fixed, and sorry!

Yesterday we were working on the KARL project, doing some post-deployment housekeeping.  Specifically, we had a checkout of the templates that had a local customization (injecting Google Analytics) that we didn’t want to check in.  At least not to the software repository itself.  We wanted building a demo KARL to have no analytics, certainly not OSI’s account.

The most logical thing would have been to throw ZPT at it and get a little snippet of HTML to jam in just before closing the body tag.  But that would mean going to a number of places and injecting calls, plus we’d have to grab the configuration data from somewhere for the right snippet.

Tres and Chris Rossi argued for middleware: something that would watch the outgoing HTML and inject Google Analytics in the appropriate circumstances.  An hour later, Tres had written repoze.urchin that parameterized in the Paste configuration file the data, then hacked the HTML on the way out.

When to use and not use middleware is an art that I’m still learning about.  The biggest two rules appear to be, don’t solve a problem with middleware if the application won’t run without it, or if the middleware requires access to information inside the application.