Giving a “Pyramid for Humans” tutorial at Plone Conf, Nov 1-2

Yep, me. Paulie. Giving a tutorial on Pyramid at the conference. Stop snickering!

Ok, sure, Dad (Chris McDonough) will be there supervising. But I’m hoping to give a whirlwind tutorial of Pyramid stuff for people like me. Civilians. It won’t be too in-depth, and it will for damn straight make sure you are successful at accomplishing each thing we cover.

The registration page has more details on who it is aimed and not aimed at, as well as the topics I hope to cover. I’ve started writing some of the material and reading some previous tutorial material from Chris and Carlos de la Guardia.

If you’re looking for a gentle but productive introduction to Python web development using Pyramid, come join us. I’m really excited about this and expect it to be a crap-ton o’ fun.

Giving a keynote at PyCon DE

Filed under “they must REALLY be scraping the bottom of the barrel to dig up ME“, I’ve been asked to give a keynote at the PyCon DE conference in Leipzig. I’m 15:30 to 16:30 on Thursday, October 6. Lightning talks are just after me, so I’m not completely the thing in between the audience and happy hour.

I used to give talks all…the…time. And now I don’t. Haven’t in what seems like a long time, unless giving a halftime talk to a team of girls lacrosse players qualifies. I’ve been out of the game for a long time and the relevance of my perspective is a bit rusty.

So I’m going to stick to perspective and look backwards. I’m collecting a bunch of moments over the last 18 years participating in steering Python/Zope/Plone/Pyramid projects. Hopefully funny, possibly poignant, certainly embarrassing. I also hope to include some anecdotes and wisdom from people I’ve met along the way who have guided projects over a long period of time.

I always used to say my talks had a content-free guarantee: if you actually learned something, I’d give you your money back. Definitely doubling down on that for this talk.

Kinda liking PyCharm

I’ve been tinkering around with IDEs for a while.  It’s hard to justify it…every time I try, I spend a tremendous amount of time using the hoped-for new tool, using it for real, only to eventually find the fatal flaws.  Then it’s back to Emacs with a dozen flowers, a box of chocolates, and the hangdog expression that says “Sorry I set my eyes on another.”

This has happened for Pydev/Eclipse, Aptana, Textmate, and Komodo.  Each brought down by some level of (a) horrible performance on OS X, (b) extreme amount of tinkering to get something working such as indentation, or (c) a realization that the cost (Komodo) meant I’d be the only one I know using.

Thought I’d give PyCharm a try.  It’s based on a larger IDE project so it automatically gains a lot of features, such as really good HTML and Javascript support.  Getting it working with virtualenv and buildout wasn’t excruciating, in fact, the added specific support for it.  And the performance was pretty good.  Considering I still can’t get decent Javascript indentation, even after manual help from others, it was a good sign.

Last week my trial run ran out.  I could have chintzed by asking them for an open source license.  But I plopped down the modest cost, expecting that I would find the fatal flaw about 10 minutes after the credit card charge cleared.

But today I looked at my screen and realized…damn, I’m pretty happy.  Still scratching the surface.  When they add reST/Sphinx support later this year, I’ll be even happier.  Maybe happy enough to put out a bounty for Chameleon/Pyramid support.

“Did you mean FOOBAR?” in PostgreSQL text indexing

This is a reminder-to-self kind of post.  Chris Rossi and I were talking about some of the “improved search” features we have planned for KARL thanks to pgtextindex and the use of PostgreSQL’s under-appreciated text indexing/retrieval story.

One thing we think we could add is “Did you mean?” support.  That is, if someone did a typo on a search term, offer some alternatives.  It’s a valuable feature (at least to me.)

The most natural thing to think about is brute-force spellchecking.  That has a number of flaws.  First, lots of things (names, for example) won’t be in any dictionary, much less some default language dictionary.  Second, you can’t show a bunch of corrections, which ones should you show?  Finally, what if you show a correction for which the word doesn’t occur in the corpus?

In Fuzzy String Matching With Trigram and Trigraphs you see how PostgreSQL helper functions can, well, help.  You can compare a term to all the reduced forms of all the words indexed in all the documents.  Then issue a query that does a soundex-style comparison to the query terms.  All using optimized indices, weighting, and scoring.  Perhaps you could also use corpus statistics to narrow the list down to words that occur in many documents.

Chris and I previously came across this when he had a neat idea on speeding up prefix searches.  Rather than do a prefix, instead find all the expansions of words/lexemes for that prefix which occur in the corpus, select the 100 most likely candidates, and do full-word searches for those 100 words.  You get better performance by omitting prefixes.  One might assert you get better quality results by letting all the full-word scoring machinery, synonyms, stemming, etc. kick in.

It’s interesting to look at some of the machinery PostgreSQL has which doesn’t get talked about much.  Like the ltree for hierarchies and tree structures.

First pypi release of repoze.pgtextindex for searching

Interested in faster/better searches in Pyramid/BFG without a change to your application, but don’t want a massive addition to your server software responsibilities?  Eager to keep transactional integrity?  repoze.pgtexindex uses the text indexing in PostgreSQL 8.4+ as a replacement for text indexing/searching in repoze.catalog’s zope.index.

Two Julys ago Shane Hathaway was working with us on the KARL project.  We had been discussing switching our KARL deployments to RelStorage.  The topic came up about text search, how important it is in our applications, and whether our search performance and quality were up to snuff.

We did some research on the text search capabilities in PostgreSQL.  Sure, it’s no Lucene or Xapian, but it looked like a significant improvement without a major change in deployment architecture.  Especially if someone was already looking at RelStorage and pgtextindex, they’d already have committed to supporting the dbms server.

That led to some experiments and we were pleased with the results.  But we’re really conservative about adding in software we can’t completely support so we put it on hold.  We then kept tinkering with it.  Chris Rossi got involved, Shane overhauled the transaction management, Chris added in implicit field weighting, etc.  We did another evaluation, then worked with Six Feet Up to make sure they were comfortable hosting/supporting RelStorage/pgtextindex/PostgreSQL.

And now we’re getting close to making the transition, so Chris worked with Shane and made a pypi release.

I’m excited about it.  Some might be attracted to big search solutions, but that’s quite a jump from the target for Zope-style applications.  I worked on a project that went big and it wasn’t exactly a pain-free drop-in.  If you’re a Pyramid/BFG person happy with zope.index then stick with it, that’s your least complex route.  But maybe you want much better search features, much faster search, half of your ZODB objects out of the ZODB and cache, and you’re already using PostgreSQL or comfortable with it.  If so, this is a much smaller step than adding a ginormous search engine and losing transactional integrity.

Some may prefer big search servers, some may prefer zope.index.  But I humbly submit that repoze.pgtextindex is another choice with certain positive qualities.

Chris McDonough on pluggable apps

We’re from the world of Zope and Plone, so we have a long history on the topic of frameworks and pluggability.  We’ve helped foist the good, the bad, and the ugly on large populations of developers over the years.  So we’re continuing to learn the right balance on frameworkiness.

Chris wrote a very good email yesterday on the Pylons mailing list on the topic of Pyramid’s pluggability story.  The “Pylons Project” wants to have a story over the long haul.  But we’re still in the mode of thinking, learning, listening, and then building.

To summarize Chris’s note (at least from my POV):

• Pyramid provides a pretty attractive extensibility story for application developers.
• Pluggability is harder, because a plug-in will need cooperating systems (storage, authentication, etc.) and Pyramid isn’t going to have opinions on all that.
• Your application will have opinions, though, thus you can provide plug points built with Pyramid’s extensibility story.
• A plugin-story’s success is directly related to the ruthlessness of the decisions and opinions being made.  The more you define the surface area by making the choices, the more viable the plug point.  Over-generalization is the enemy.
• The Pylons Project would like to make an application on Pyramid that has opinions, provides higher-level services, and thus has a strong plugin story.

At the end of the day, the Python world has a lot of great alternatives already for web frameworks.  Anything new that tries to emerge needs fresh ideas.  I feel like extensibility and pluggability are still areas ripe for new thoughts.

While many might want less (e.g. single file microframeworks) but there might be a constituency for those that want more.

Wonderful blog post on Pyramid development

I’ve wanted for a while to write an introductory piece on Pyramid.  That is, how to use Pyramid to do a style of development that I find very simple and intuitive: traversal and ZODB.

Alex Marandon wrote exactly the article I wished I would have written.  He doesn’t presume you know Zope.  He then takes the (from my POV, super-easy) ideas of traversal and persistence, and introduces them in a simple, obvious way.

I like jQuery UI

Over the last year I’ve done more work with jQuery and specifically, widget frameworks atop it.  I’ve spent most of my time with jQuery UI and jQuery Tools.  I like them both, but I wound up settling on the former and spending quite a bit of time with it.

Thus I’ve watched with some amount of curiosity an anti-jQuery-UI meme emerging in some projects I’ve participated in.  It isn’t just that they favor jQuery Tools.  Rather, they also seem intent on some broad-brushed jQuery UI bashing to go with it.  Most of the time the word “bloat” emerges, and I thought I’d look into it.

“Bloat” is usually something that can be easily measured.  For example, ahem, some open source content management systems ship with a significant amount of code, thus a lot of “bloat”.  But when looking at jQuery UI (jqUI) vs. jQuery Tools (jqT), I found:

• jqUI: 19 Kb
• jqT: 2.4 Kb

That’s the size of the JS for each, gzip’d (in the way any normal production webserver would deliver it.)  For the comparison, I made a jQuery UI download that matches the same widget framework features used by Plone, to make it even.  It isn’t utterly apples-to-apples: jQuery UI also has CSS and images which add marginally to the Kb needed on a site.

But just to put this into perspective, the difference is about 1/2 the size of the Plone logo.  I don’t think “bloat” could apply to pageload impact, but even if it did…who cares?  Any intelligent site uses far-future expires and versioned URLs to ensure visitors only need one request for the JS library…ever.

So perhaps “heavy” and “bloat” don’t mean measurable things.  Perhaps the point is conceptual heaviness.  It’s true that the jQuery UI widget framework adds some ceremony in the constructor and method dispatch.  But two notes in its defense.

First, the method dispatch pattern is the one recommended by the jQuery core team for plugin extensibility.  Well, I’m sure “jQuery core team” probably includes lots of viewpoints, but the page I read on the topic seemed authoritative.

Second, there is a reason the ceremony is there: widget re-use and extensibility.  There’s always a price to be paid for this.  But when building a big system, the price can be worthwhile.  I suspect systems like Plone will be paying that price at some point, just with perhaps different ceremony.

I’ve seen “bloat” connected to “CSS bloat”.  I’m also not sure what specifically that means, I haven’t seen any comparisons to jqT selector hierarchies.  I certainly can sympathize with this point: the jQuery UI CSS Framework has lots of classes for things such as .ui-widget and .ui-widget-container and .ui-widget-content.  But so does Plone’s templating, and for the same reason as the above: large systems need fine-grained extensibility and customizability.

Personally I’m happy jqUI went that way, as they are busy stripping their main widgets down dramatically, composed from a small set of building blocks that get extended.  Even better, they documented their selector hierarchy in a fairly-visible contract.

At the end of the day, my best guess is that “bloat” is being used in a non-measurable way as a style choice.  Which is absolutely fine.  It’s great to say “I prefer jQuery Tools”.  It’s great that both exist, as I think they target different audiences.  jQuery Tools when you want to quickly get something done without learning a lot of ceremony, jQuery UI when you have a big system that needs a focus on extensibility and customization.

Maybe I have it wrong, and “bloat” is something specific and empirical.  Either way, they are both good choices and fairly similar in features, size, and support.

Pylons and BFG, sitting in a tree, k-i-s-s-i-n-g

First, repoze.bfg had its long-awaited 1.3 release last week.  For fans of the lightweight framework, this release was a nice wrapping-up point.  i18n/l10n and lots of little tweaks to scratch itches reported by developers.  Not only was Chris fast getting those wishes out the door, he did his usual exceptional job on getting the docs up-to-date.

As it turned out, getting 1.3 out the door was important.  It’s the last major release of BFG, which is now in maintenance mode, because…

…Pylons and repoze.bfg are merging!  First, the projects are merging.  Ben Bangert and Chris McDonough as project leads, plus the core developer teams for each project, have combined forces into the Pylons Project.  And the first deliverable is already evaluation-ready.  BFG 1.3 has been renamed Pyramid and will serve as the lightweight web framework for other activity in the Pylons Project.

Because it is based on BFG, Pyramid gains 600+ pages of docs, 100% test coverage, and many other qualities under the banner of “Small, Fast, Documented, Tested, Stable, Friendly, Extensible.”  In addition, Pyramid adds new machinery BFG to fulfill Pylons 1.x style programming.

Both repoze.bfg 1.3 and Pylons 1.x will continue a long period of bug fixing.  Applications that don’t want the change, but want maintenance, will thus be pleased.

I had a chance to meet with Ben, Mark Ramm, Chris, and Chris Rossi in Vegas a few weeks ago to talk about all of this.  It was an exceptional trip.  So often you see pride and egos in open source projects cause forking and splintering.  In this case, projects did the opposite: join forces to maximize how many resources are available for producing something of long-term quality.

Moreover, it’s fun working with new people and getting fresh ideas.  The problems that were solved 3 years ago in the last big cycle of Python web frameworks aren’t the same as the problems to solve in the next 3 years.  I honestly believe the Pylons Project will have some fresh ideas to bring to the table, as well as a commitment to meat-and-potatoes excellence in documentation, testing, support, speed, etc.

Developers from different communities (Pylons, Zope, etc.) have a new choice, one with momentum, very long experience in writing and supporting large applications, and a good group of people involved.  I’m looking forward to the next 3 years.

Lifestyle Inc. Redux

I was just reading the John Sculley interview about Steve Jobs.  Like others, I was deeply impressed at the humility shown by Sculley.  Not just faux humility, but actual humility, pulling no punches about how things went down and admitting that his way wasn’t the right way.

That part was particularly interesting to me.  I’ve long had difficulty with “there’s a right and a wrong”.  Like most Americans, I want to believe in silver bullets, miracle cures, and simple answers to hard problems.  There is virtue in that, and many cases, doing something questionably right but with conviction is indeed a better course than sitting on the sidelines.  As a (mediocre) engineer and a former Navy officer, you’re trained to think problems have solutions, or more correctly, solution (singular).  One true path, follow it to victory.

In reading Sculley’s comments on Jobs, I got wrapped around this axle again.  It is tempting to think “Jobs did things the right way”, but clearly that’s wrong.  The conventional wisdom at the time defined the Right Way, and (later) VCs existed to sort out the winners and losers according to the scorecard.  But the right way was, in this case, wrong.

In fact, there wasn’t “a” way at all.  There might have been many ways, right and wrong, but none of that matters.  Instead, whatever Jobs did was his way, and since it succeeded, was the right way, even though that way was completely “wrong” according to the thought leaders.  There wasn’t a way that he did.  Instead, he just did things, and since it worked, it was a way and it was a right way.

That’s just indigestible for my constitution.  I’m so wired to find paths, pick the right one, and follow it to the successful destination, that it’s hard to countenance that the right path was whichever one worked.

But looking back on it, I realize I learned this a while back.  6 years and 8 months ago, as of today, Chris McDonough wrote Lifestyle Inc.  It was an odd time for me in 2004, possibly for Chris as well.  I had just gotten off a track where there was a scorecard, a right way, and success.  And conversely, a wrong way which, if it didn’t measure against that right scorecard, was a failure.

Chris’s blog post, as the comments show, struck a lot of people.  I know many, many folks who read it and really, deeply, poignantly got it.  At the time, it gave me a nice catchy phrase to express and connote what I was going to do.

But Lifestyle Inc. and the companion Agendaless Consulting (and the backstory that goes with it) started teaching me some other lessons related to the Jobs article.  As I’ve gotten to work with Chris and Tres over the years, I’ve repeatedly seen my fervent desire to be told the right answer to follow, prove to be misguided.  I also learned from Jim Fulton on this point, who was always much better at patience and reality.  Much of the time, there isn’t a right way.  If you do something, and it works, it was the right way.

Which kind of ties a bow around the Jobs article and Lifestyle Inc.  Do the right thing for you, and do it well, and let Darwin decide if it was the right way or not.