Python virtualenvs: Inside project or outside?

November 25, 2015

I’m a dummy. I pulled the old “pretend to ask an honest question but really to confirm your belief” trick and, shockingly-not, got refuted. My small sample seems to like making their Python virtual environments outside of their projects.


I do a lot of Python explaining (Py-splaining?) and, contrary to the “see how much our Won-The-War framework can do in minus ten lines of hello”, I try to suck it up and show the right way. This usually means, make a virtual environment. It also means make a package, which ranks up there with “drop a frozen turkey in a deep fryer using your teeth” on the scale of holiday fun.

Virtual environments are a little squirrelly to explain. Like “mobile first” for web dev, I have committed to Py3-first for teaching, which means you get pyvenv for free. I then show making a and doing pip install -e . (or python ./ develop), binding that virtualenv to this project.

In my mind, that means the virtualenv is part of this project directory. If I give up on the project, I delete the directory, and everything goes with it. If I’m using SQLite, I store my .sqlite file in that working directory as well. If I’m doing JS frontend stuff, my node_modules is in that directory. It thus makes sense that my virtualenv (or plural, if I’m doing multiple Python versions) should be in the same working directory. It’s an artifact of that project.

In fact, Python’s “Hitchhiker’s Guide to Python” advertises this workflow.

I posed the question on Twitter because PyCharm, when making a new project, wants the virtualenv outside of the to-be-created project directory. I was looking to bolster my viewpoint.

The Herd Is Wrong

The wisdom of crowds voted against me 12-6. That’s the trouble with crowds, they are only wise when they agree with me. I guess that’s the difference between a crowd and a mob.

I think I’ll march on, teaching to firmly associate the virtualenv with the project, holding up the Hitchhiker’s Guide as my certificate of authenticity. But I won’t file a likely-disruptive ticket in PyCharm.

My O’Reilly video series on Pyramid is up

November 24, 2015

Earlier this year, just before PyCon, I talked to O’Reilly about doing a Pyramid video course in their commercial training catalog. It’s now in “early access”, titled “Web Applications with Python and the Pyramid Framework“.

What an effort it turned out to be. I’m certainly grateful that it has wrapped up, but I’m also glad I did it. I put a serious amount of work (months) into writing, re-writing, and re-re-rewriting the narrative and the examples. Thanks go out to Chris and Tres for steering me throughout on best practices, to include the admonition that chasing best practices is a fool’s errand.

Several items of reflections:

  1. I resisted the urge to go off the beaten path. It’s routes, Jinja2, and SQLAlchemy. I originally had “traversal in SQL” in the proposal. I chopped it out, despite my ongoing attempt at “pyramid_sqltraversal” (on GitHub.)
  2. I put more emphasis than I expected on the JSON section and frontend web applications.
  3. I also put more into extensibility (aka “framework framework”) than I had planned. I’m not that good at that story, so perhaps that’s why… I finally had a reason to learn it. And damn, Pyramid’s awesome at that.
  4. Ditto for authentication and authorization combined with SQLAlchemy. I now know a lot more about how to do that, and it’s a great story. As an aside, root/route factories (and thus having a context) should be the way we teach people to start.
  5. Of course it featured PyCharm. I couldn’t have known that, as I wrapped up, I’d be talking to JetBrains about joining the PyCharm team.
  6. I got pretty good at Camtasia, geezus.
  7. Python’s packaging story is such a bear to explain to novices.

So six hours of (unedited) video later, it’s on the way out the door. Pyramid is so unique and so powerful…hope this series helps some mortals who are trying to get started.

PyRVA meet up and type hinting

November 18, 2015

Python’s community has long been considered as valuable as the
software, and that extends to Python local meetups. They’re fun to
attend, but what’s even more fun is watching one during its formation.

Last week I attended the second PyRVA meet up in Richmond, VA and talked about Python 3.5 type hinting in PyCharm. The user group is founded by Andrew Elbert. The location was part of Virginia Commonwealth University (VCU), a large public university in the heart of Richmond. Really nice meet up space: table seating for everyone and humongous display for presentation.

I’ve been to a lot of these, over the decades and across continents. I seem to enjoy the people in this business more than the software, so I have a fascination for how people group-up and do new things. This meetup was interesting in several ways:

  • Diverse in ethnicity, gender, age and skill level
  • First half was bootcamp-style, with experienced people working with
  • Part “CS student learning another language” and part “I have a job but want to advance my career by learning programming skills”
  • It’s a testament to the founders that they have injected this vibe in from the beginning.

The evening started with sandwiches, yummy. The room filled up, with people sitting around the sides, so almost 40 people, and this was the only the second session. Clearly they have tapped into something. The bootcamp part ran for about an hour. Then a talk on devops with Jenkins, followed by a fun talk on using Python to analyze the historical reading-level of Jeopardy questions.

I talked about type hinting in Python 3.5, as covered in an article I am finishing up this week. I also covered general questions about PyCharm. I’m learning more about the target audience, for IDEs in general and PyCharm specifically, and this was a good chance to again talk to such-minded people. One thing I’m learning: some beginners are more comfortable seeing “Python” on the screen. Not a terminal, with an editor they need to run and a shell to run a command, etc. They want something that frames their new world and gives them targets to interact with.

And of course, like all my talks, it was just a thin excuse to give two decades of Python-related funny stories.

PyRVA is one I will keep an eye on. I hope they grow, keep the enthusiasm up, and preserve the unique qualities they have started with.

New adventure, PyCharm Developer Advocate at JetBrains

October 14, 2015

At the start of October, I began a new adventure. I’m the PyCharm and WebStorm Developer Advocate at JetBrains. More of the former, some of the latter. This morning JetBrains published the introductory interview with more detail.

I’m super excited about this. As I said in the article, I’ve been wanting for a while to work with others in a product company, learning about all the ways stuff really gets made. And lucky me, the product I use the most, PyCharm, was looking for someone to do the kind of job I enjoy the most.

For now I’ll still be doing a portion of time with Agendaless on existing work. And of course, I’ll still be in the communities I’m currently in, probably doing more, as JetBrains has legit support for people spending time on open source work. It’s deeply, deeply exciting to think of all the relationships in Python I’ll be working on in the coming years. Lots of good going on that we can support.

Faster relevance ranking didn’t make it into PostgreSQL 9.4

October 29, 2014

Alas, the one big feature we really needed, the patch apparently got rejected.

PostgreSQL has a nice little full-text search story, especially when you combine it with other parts of our story (security-aware filtering of results, transactional integrity, etc.) Searches are very, very fast.

However, the next step — ranking the results — isn’t so fast. It requires a table scan (likely to TOAST files, meaning read a file and gunzip its contents) on every row that matched.

In our case, we’re doing prefix searches, and lots and lots of rows match. Lots. And the performance is, well, horrible. Oleg and friends had a super-fast speedup for this ready for PostgreSQL 9.4, but it apparently got rejected.

So we’re stuck. It’s too big a transition to switch to ElasticSearch or something. The customer probably should bail on prefix searching (autocomplete) but they won’t. We have an idea for doing this the right way (convert prefixes to candidate full words, as Google does, using PG’s built-in lexeme tools) but that is also too much for budget. Finally, we don’t have the option to throw SSDs at it.

Recent work with RelStorage

October 5, 2014

The KARL project has been focused in the last year on some performance and scalability issues. It’s a reasonably big database, ZODB-atop-RelStorage-atop-PostgreSQL. It’s also heavily security-centric with decent writes, so CDNs and other page caching wasn’t going to help.

I personally re-learned the ZODB lesson that the objects needed for your view better be in memory. Our hoster is 32-bit-only, so that made us learn a little more and think about the tradeoffs. Adding more threads means higher memory usage per process. We get some concurrency as PG requests release the GIL, but that’s hard to bank on. Instead you have a bunch of single-connection processes, but then you get little cache affinity. One bad view leads to 20k objects getting requested. The user hits reload a few times, and your site is hung until PG is finished.

We use memcache as well, but that is also limited to 2 GB. We likely need to spread the cache across a few processes.

We decided to do some timing of various scenarios, with the hardware and dataset that we had. A completely cold PG server (nothing in OS read buffers), a warm PG, stuff in memcache, stuff in the RelStorage local cache, and stuff in the ZODB connection cache. Here is what Shane found:

  • 179 seconds when PG has to fetch a lot of the data from disk.
  • 21 seconds when PG has all the data in its own cache.
  • 6 seconds when memcache has all the pickles.
  • 2 seconds when the local client cache has all the pickles.
  • 1.2 seconds when the ZODB cache is filled.

We had previously thought that unpickling was a bottleneck.  It wasn’t. We then did some research on Python standard library compression at the lowest level. Turns out that we can get decent compression at very high speeds. So we thought about RelStorage’s off-overlooked “local cache”, an in-memory, process-wide pickle cache. By default it is set to a low number.

The local cache had an enticing aspect: the code was tinkerable, as it was under Shane’s control. What if we could play with some ideas? For example, only cache objects under a certain size. Some big PDF might take up the space of thousands of (far more important) catalog objects. RelStorage gained a knob for setting a size threshold. Compression, though, is very interesting. With it we can get many more objects in the cache, and not pay too much of a price.

Both of these (size limit and compression level) are now in RelStorage. It will take a while to decipher the right combination of ZODB connections vs. client cache vs. local cache, and which numbers to up/down before hitting 2 GB range. But we’ve already had a big impact on performance.

And a plug for some other work that the KARL project funded Shane for….packing improvements that went into b2 a couple of days ago, plus perfmetrics decorators that let you spew all kinds of ZODB-oriented stats to graphite/DataDog.

Plone runs Brazil

September 28, 2013

Plone, baby, you still got the power to amaze.

I saw this tweet by André Nogueira (@agnogueira):

And it’s true. The President of Brazil announced the web portal, and at the bottom, you see that it is powered by Plone. I’m at National Airport, on the way to Atlanta, on the way to Brasilia for the conference. And that’s a nice thing to see.

Me on the web, 20 years ago today

September 2, 2013

September 2, 1993 is my first recorded post about the web, asking a question answered by Marc Andreesen.

As backstory, I was a Navy officer waiting around to start flight school. I was part of the “Top Gun” generation: in 1986, lots of silly young men go watch Top Gun, then four years later the Navy has an incredible surplus of pilots. Great foresight there, Navy. For over a year I got paid to go shark fishing in Pensacola, which really was as awesome as it sounds. Thanks, taxpayers!

Ultimately my eyesight went over the allowed limit before my start date arrived, so I had to find something else to do in early 1992. As it turns out, Pensacola was home to the “Navy Internet Manager”. I transferred over to that group with a task of getting some valuable services onto that crazy Internet thing. Email, telnet, DNS, etc. Then gopher. Then the big argument. gopher+ versus this new WWW thing.

The guy in the cubicle next to me ran the DNS domain. I vividly remember, in early 1993, rolling my chair around the partition wall and asking for “What’s that?” he asked. Hilarity ensues. I’ll talk about my experience with that in another post next month.

New blog posts: Chris, Tres, me

August 12, 2013

Last week, Tres and I wrapped up a little website for Agendaless, based on Pyramid and Substance D. Not very ambitious site, but still a lot of fun working in Substance D.

We put up some blog posts:

  • Chris wrote a really thought-provoking article about our experiences when open source and consulting rub up against the sad state of patent shakedowns. Someone like Chris writes and gives away a multi-million-dollar code base. When it is time to do some consulting, he’s asked to indemnify the small, custom against patents as part of the contract.  The yearly insurance cost is over half his yearly income.
  • Agendaless is turning 7 in a couple of weeks. Tres did a little retrospective.
  • 11 years ago I moved to France, with a focus soon thereafter on large-scale project management.

PyCharm 3.0 EAP2 supports Pyramid

August 2, 2013

Confession: I’m an IDE user. Years of Emacs with occasional flirtations with IDEs (Komodo, PyDev) always led me back to Emacs. PyCharm (with the requisite 4 GB RAM upgrade for the Java tax) is the one that stuck.

PyCharm has never had any specific Pyramid support, until now. The upcoming 3.0 version will have some support for Pyramid, and the latest early access preview makes this available. Here is a screencast I just made demonstrating the support:

Screencast showing Pyramid support in PyCharm 3.0 EAP

In this demo I show:

  • Creating Project using “Project type” of Pyramid
  • Making a virtual environment under Python 3.3.2 (using pyvenv) for the new project
  • Telling PyCharm to install Pyramid into the project
  • Choosing one of the Pyramid-provided “scaffolds” to generate a working sample
  • Using the PyCharm-generated “Run Configuration” to easily execute the generated
  • Using the PyCharm-generated “Run Configuration” to start the Pyramid project and view the home page in a browser
  • I also show a reminder to bring in stuff for the office omelets (my 3 chickens are laying eggs faster than we can eat them)

Two caveats I covered:

  • There is a bug, now fixed, in the run configuration about working directory. The next EAP should include the fix.
  • If you want the scaffold to generate a sample, it is important to click “No” when PyCharm warns you about an existing directory. (This only happens if, like I do, you use virtual environments stored in the project directory.)

I don’t know how much other support JetBrains plans for Pyramid in PyCharm. I doubt they’ll get Chameleon support, for example. But for those trying to get started quickly, this really eliminates a lot of monkey business related to virtual environments, getting easy_install/pip into the virtualenv, getting the sources and running a scaffold, etc.


Get every new post delivered to your Inbox.

Join 590 other followers