Off to PyCon 2017

On the train to the plane to the plane to the whatever to the hotel to meetup up with the PyCharm gang for PyCon. Like everybody else, I measure the passage of time by PyCons and look forward to old friends, familiar faces, and meeting new people.

We’re getting a gang of folks (Guido, Barry, Jim, me) from the original NIST workshop in 1994 together for the keynote panel on Sunday morning. What was going through our heads back then? What did Python do right and wrong? If you have any questions, come by the PyCharm booth, or even better, ask during the panel. It’s going to be interactive and a whole barrel-o’-fun.

Yes, I’m posting this on my ancient WordPress blog. I’m working on a replacement in Sphinx, teaching myself Sphinx extensions and brushing up on Bootstrap 4 etc. But that project is mega-months behind. I’m also experimenting with Hugo on a local girls lacrosse site (stafflax) hosted on Firebase. It’s been a fun learning experience.

Advertisement

Making the material for the Polyglot webinar

Tomorrow we do Part 2 of our “Polyglot Python with PyCharm” webinar, talking again about modern web frontend development, from a Python perspective, within PyCharm. We put a lot of work into writing materials for this. Why, and what’s next?

In the first webinar, I spent a lot of advance time breaking down the session into a series of steps, building a ToDoMVC-style application with a Flask-Restless backend. I wrote the code, then near the end, found a fatal flaw in the “pedagogy” of the sequence, and had to rewrite it. Only to find another, repeat.

Near the deadline, I decided to put everything in a publicly-visible repo, so webinar viewers could follow along. At the very last second, I put the notes I was using to guide my progression, into the repo as well.

Two unexpected outcomes…first, I ran out of time. In fact, I ran out of time at the point where I was showing the really cool part that everything had led up to. Next, the feedback showed that registrants really wanted the material, with explanations, in advance.

I had been planning for a while to write a series for this. This crazy frontend world is hard to decipher for Python folks, but we know we need to jump in. I wanted to be a guide, explaining this world in Python terms, and showing how to couple it to Python-based backends. Also, if you get fully into that world, you’re better off with IDE help, and PyCharm is *very* unique in that regard: first-class Python and first-class JS/HTML/CSS support. In fact, many of my Python friends didn’t know PyCharm could do what WebStorm does.

So I used this second part of the webinar to write up all the story for this “Polyglot Python with PyCharm”. It is a series of articles on each part of the equation, followed by a start-to-finish ToDoMVC application. For those familiar with my “Quick Tutorials” for Pyramid, it’s that kind of approach. Individual slices, with a lot about the “why”, plus working code.

Here’s the (current) table of contents. For the material, register for the webinar and you’ll get the link, or wait until next week:

• NodeJS for Python and PyCharm Developers
• JavaScript Packaging with package.json
• Linting with ESLint
• Automation with npm run Scripts
• Modules with CommonJS
• Browser Bundling with Webpack
• Browser Debugging from PyCharm
• TDD with Mocha
• ES6 Imports with Babel
• Pythonic JavaScript with ES2015
• DOM TDD with JSDOM
• Initial ToDoMVC
• Frontend Tooling for ToDoMVC
• Moving the Frontend Out of the Backend
• Bundling ToDoMVC with WebPack
• ES6 Imports for ToDoMVC
• Pythonic JavaScript for ToDoMVC
• TDD for ToDoMVC

I still have more to do to finish this after this week: fill in the missing writeup on the ToDo app, do a tenth-round of rewriting, and make a screencast video for each article. Also, it’s tied to a new blog for me which isn’t yet really official.

But in general, we think PyCharm has fantastic story for Python-backed frontend development, and this series is a commitment to being your guide. Hope to see you tomorrow in the webinar.

Javascript development, in a Python IDE?

I hear this a lot: “I use PyCharm for Python, do I need to get WebStorm to do web front-ends?” It makes sense that people would say this: PyCharm is for Python, WebStorm is for HTML/CSS/JS.

Thing is, PyCharm has the same engine inside it that WebStorm has (and vice versa.) You can do WebStorm stuff, in PyCharm.

For example, let’s run some JavaScript:

Look at that…JavaScript running in PyCharm via /usr/local/bin/node. Same way you do it in WebStorm. How can that be?

Turns out PyCharm has the IntelliJ plugins that do the JS/HTML/CSS stuff, and much of it is wired into the PyCharm UI. The gif above shows how it is exposed into PyCharm preferences, which let you choose your interpreter and install packages, just like with Python interpreters in PyCharm.

Just like with Python, this creates a Node-based “Run Configuration”:

That’s all well and good, but surely there’s no JavaScript debugger in PyCharm?

Webinar Tomorrow, Tuesday Jan 12

Want to see more? There’s more to show, and tomorrow’s PyCharm webinar is the start. I’m really looking forward to the webinar and the subsequent material.

I’m doing a webinar next week: “Polyglot PyCharm for Rich Web Frontends”

Next Tuesday at 10AM Eastern I’m doing a free/live JetBrains webinar. Nutshell: using the WebStorm-like web features, in PyCharm.

This webinar originated from several of my Python friends asking if they should get WebStorm for their web frontend development to go with PyCharm for their backend Python development. They were surprised to hear that PyCharm inherits many of the HTML/CSS/JS platform features from WebStorm.

Also, over the last few years, I’ve done quite a lot with the modern frontend stack: NodeJS, npm, grunt/gulp, Babel for ES6, Webpack, Angular, etc. In particular, I’ve approached that world as an émigré from Pythonistan, and thus adopted some workflows that seemed more Pythonic. At the same time, I talk with my grumpy old Python friends who want modern JS MVC to get off their lawn. I’d like to be a friendly tour guide for Python folks, to explain and interpret this clown car..ahem, brave new world, but in terms familiar to them.

This webinar is the first step, and I hope to provide more on “Polyglot Python” via PyCharm in the coming months.

Python virtualenvs: Inside project or outside?

I’m a dummy. I pulled the old “pretend to ask an honest question but really to confirm your belief” trick and, shockingly-not, got refuted. My small sample seems to like making their Python virtual environments outside of their projects.

Background

I do a lot of Python explaining (Py-splaining?) and, contrary to the “see how much our Won-The-War framework can do in minus ten lines of hello”, I try to suck it up and show the right way. This usually means, make a virtual environment. It also means make a package, which ranks up there with “drop a frozen turkey in a deep fryer using your teeth” on the scale of holiday fun.

Virtual environments are a little squirrelly to explain. Like “mobile first” for web dev, I have committed to Py3-first for teaching, which means you get pyvenv for free. I then show making a setup.py and doing pip install -e . (or python ./setup.py develop), binding that virtualenv to this project.

In my mind, that means the virtualenv is part of this project directory. If I give up on the project, I delete the directory, and everything goes with it. If I’m using SQLite, I store my .sqlite file in that working directory as well. If I’m doing JS frontend stuff, my node_modules is in that directory. It thus makes sense that my virtualenv (or plural, if I’m doing multiple Python versions) should be in the same working directory. It’s an artifact of that project.

In fact, Python’s “Hitchhiker’s Guide to Python” advertises this workflow.

I posed the question on Twitter because PyCharm, when making a new project, wants the virtualenv outside of the to-be-created project directory. I was looking to bolster my viewpoint.

The Herd Is Wrong

The wisdom of crowds voted against me 12-6. That’s the trouble with crowds, they are only wise when they agree with me. I guess that’s the difference between a crowd and a mob.

I think I’ll march on, teaching to firmly associate the virtualenv with the project, holding up the Hitchhiker’s Guide as my certificate of authenticity. But I won’t file a likely-disruptive ticket in PyCharm.

My O’Reilly video series on Pyramid is up

Earlier this year, just before PyCon, I talked to O’Reilly about doing a Pyramid video course in their commercial training catalog. It’s now in “early access”, titled “Web Applications with Python and the Pyramid Framework“.

What an effort it turned out to be. I’m certainly grateful that it has wrapped up, but I’m also glad I did it. I put a serious amount of work (months) into writing, re-writing, and re-re-rewriting the narrative and the examples. Thanks go out to Chris and Tres for steering me throughout on best practices, to include the admonition that chasing best practices is a fool’s errand.

Several items of reflections:

1. I resisted the urge to go off the beaten path. It’s routes, Jinja2, and SQLAlchemy. I originally had “traversal in SQL” in the proposal. I chopped it out, despite my ongoing attempt at “pyramid_sqltraversal” (on GitHub.)
2. I put more emphasis than I expected on the JSON section and frontend web applications.
3. I also put more into extensibility (aka “framework framework”) than I had planned. I’m not that good at that story, so perhaps that’s why… I finally had a reason to learn it. And damn, Pyramid’s awesome at that.
4. Ditto for authentication and authorization combined with SQLAlchemy. I now know a lot more about how to do that, and it’s a great story. As an aside, root/route factories (and thus having a context) should be the way we teach people to start.
5. Of course it featured PyCharm. I couldn’t have known that, as I wrapped up, I’d be talking to JetBrains about joining the PyCharm team.
6. I got pretty good at Camtasia, geezus.
7. Python’s packaging story is such a bear to explain to novices.

So six hours of (unedited) video later, it’s on the way out the door. Pyramid is so unique and so powerful…hope this series helps some mortals who are trying to get started.

PyRVA meet up and type hinting

Python’s community has long been considered as valuable as the
software, and that extends to Python local meetups. They’re fun to
attend, but what’s even more fun is watching one during its formation.

Last week I attended the second PyRVA meet up in Richmond, VA and talked about Python 3.5 type hinting in PyCharm. The user group is founded by Andrew Elbert. The location was part of Virginia Commonwealth University (VCU), a large public university in the heart of Richmond. Really nice meet up space: table seating for everyone and humongous display for presentation.

I’ve been to a lot of these, over the decades and across continents. I seem to enjoy the people in this business more than the software, so I have a fascination for how people group-up and do new things. This meetup was interesting in several ways:

• Diverse in ethnicity, gender, age and skill level
• First half was bootcamp-style, with experienced people working with
  beginners
• Part “CS student learning another language” and part “I have a job but want to advance my career by learning programming skills”
• It’s a testament to the founders that they have injected this vibe in from the beginning.

The evening started with sandwiches, yummy. The room filled up, with people sitting around the sides, so almost 40 people, and this was the only the second session. Clearly they have tapped into something. The bootcamp part ran for about an hour. Then a talk on devops with Jenkins, followed by a fun talk on using Python to analyze the historical reading-level of Jeopardy questions.

I talked about type hinting in Python 3.5, as covered in an article I am finishing up this week. I also covered general questions about PyCharm. I’m learning more about the target audience, for IDEs in general and PyCharm specifically, and this was a good chance to again talk to such-minded people. One thing I’m learning: some beginners are more comfortable seeing “Python” on the screen. Not a terminal, with an editor they need to run and a shell to run a command, etc. They want something that frames their new world and gives them targets to interact with.

And of course, like all my talks, it was just a thin excuse to give two decades of Python-related funny stories.

PyRVA is one I will keep an eye on. I hope they grow, keep the enthusiasm up, and preserve the unique qualities they have started with.

New adventure, PyCharm Developer Advocate at JetBrains

At the start of October, I began a new adventure. I’m the PyCharm and WebStorm Developer Advocate at JetBrains. More of the former, some of the latter. This morning JetBrains published the introductory interview with more detail.

I’m super excited about this. As I said in the article, I’ve been wanting for a while to work with others in a product company, learning about all the ways stuff really gets made. And lucky me, the product I use the most, PyCharm, was looking for someone to do the kind of job I enjoy the most.

For now I’ll still be doing a portion of time with Agendaless on existing work. And of course, I’ll still be in the communities I’m currently in, probably doing more, as JetBrains has legit support for people spending time on open source work. It’s deeply, deeply exciting to think of all the relationships in Python I’ll be working on in the coming years. Lots of good going on that we can support.

Faster relevance ranking didn’t make it into PostgreSQL 9.4

Alas, the one big feature we really needed, the patch apparently got rejected.

PostgreSQL has a nice little full-text search story, especially when you combine it with other parts of our story (security-aware filtering of results, transactional integrity, etc.) Searches are very, very fast.

However, the next step — ranking the results — isn’t so fast. It requires a table scan (likely to TOAST files, meaning read a file and gunzip its contents) on every row that matched.

In our case, we’re doing prefix searches, and lots and lots of rows match. Lots. And the performance is, well, horrible. Oleg and friends had a super-fast speedup for this ready for PostgreSQL 9.4, but it apparently got rejected.

So we’re stuck. It’s too big a transition to switch to ElasticSearch or something. The customer probably should bail on prefix searching (autocomplete) but they won’t. We have an idea for doing this the right way (convert prefixes to candidate full words, as Google does, using PG’s built-in lexeme tools) but that is also too much for budget. Finally, we don’t have the option to throw SSDs at it.

Recent work with RelStorage

The KARL project has been focused in the last year on some performance and scalability issues. It’s a reasonably big database, ZODB-atop-RelStorage-atop-PostgreSQL. It’s also heavily security-centric with decent writes, so CDNs and other page caching wasn’t going to help.

I personally re-learned the ZODB lesson that the objects needed for your view better be in memory. Our hoster is 32-bit-only, so that made us learn a little more and think about the tradeoffs. Adding more threads means higher memory usage per process. We get some concurrency as PG requests release the GIL, but that’s hard to bank on. Instead you have a bunch of single-connection processes, but then you get little cache affinity. One bad view leads to 20k objects getting requested. The user hits reload a few times, and your site is hung until PG is finished.

We use memcache as well, but that is also limited to 2 GB. We likely need to spread the cache across a few processes.

We decided to do some timing of various scenarios, with the hardware and dataset that we had. A completely cold PG server (nothing in OS read buffers), a warm PG, stuff in memcache, stuff in the RelStorage local cache, and stuff in the ZODB connection cache. Here is what Shane found:

• 179 seconds when PG has to fetch a lot of the data from disk.
• 21 seconds when PG has all the data in its own cache.
• 6 seconds when memcache has all the pickles.
• 2 seconds when the local client cache has all the pickles.
• 1.2 seconds when the ZODB cache is filled.

We had previously thought that unpickling was a bottleneck.  It wasn’t. We then did some research on Python standard library compression at the lowest level. Turns out that we can get decent compression at very high speeds. So we thought about RelStorage’s off-overlooked “local cache”, an in-memory, process-wide pickle cache. By default it is set to a low number.

The local cache had an enticing aspect: the code was tinkerable, as it was under Shane’s control. What if we could play with some ideas? For example, only cache objects under a certain size. Some big PDF might take up the space of thousands of (far more important) catalog objects. RelStorage gained a knob for setting a size threshold. Compression, though, is very interesting. With it we can get many more objects in the cache, and not pay too much of a price.

Both of these (size limit and compression level) are now in RelStorage. It will take a while to decipher the right combination of ZODB connections vs. client cache vs. local cache, and which numbers to up/down before hitting 2 GB range. But we’ve already had a big impact on performance.

And a plug for some other work that the KARL project funded Shane for….packing improvements that went into b2 a couple of days ago, plus perfmetrics decorators that let you spew all kinds of ZODB-oriented stats to graphite/DataDog.