Python virtualenvs: Inside project or outside?

I’m a dummy. I pulled the old “pretend to ask an honest question but really to confirm your belief” trick and, shockingly-not, got refuted. My small sample seems to like making their Python virtual environments outside of their projects.

Background

I do a lot of Python explaining (Py-splaining?) and, contrary to the “see how much our Won-The-War framework can do in minus ten lines of hello”, I try to suck it up and show the right way. This usually means, make a virtual environment. It also means make a package, which ranks up there with “drop a frozen turkey in a deep fryer using your teeth” on the scale of holiday fun.

Virtual environments are a little squirrelly to explain. Like “mobile first” for web dev, I have committed to Py3-first for teaching, which means you get pyvenv for free. I then show making a setup.py and doing pip install -e . (or python ./setup.py develop), binding that virtualenv to this project.

In my mind, that means the virtualenv is part of this project directory. If I give up on the project, I delete the directory, and everything goes with it. If I’m using SQLite, I store my .sqlite file in that working directory as well. If I’m doing JS frontend stuff, my node_modules is in that directory. It thus makes sense that my virtualenv (or plural, if I’m doing multiple Python versions) should be in the same working directory. It’s an artifact of that project.

In fact, Python’s “Hitchhiker’s Guide to Python” advertises this workflow.

I posed the question on Twitter because PyCharm, when making a new project, wants the virtualenv outside of the to-be-created project directory. I was looking to bolster my viewpoint.

The Herd Is Wrong

The wisdom of crowds voted against me 12-6. That’s the trouble with crowds, they are only wise when they agree with me. I guess that’s the difference between a crowd and a mob.

I think I’ll march on, teaching to firmly associate the virtualenv with the project, holding up the Hitchhiker’s Guide as my certificate of authenticity. But I won’t file a likely-disruptive ticket in PyCharm.

Advertisements

15 Responses to “Python virtualenvs: Inside project or outside?”

  1. tiarno Says:

    I’m with you. Stupid mob.

  2. Grzegorz Śliwiński Says:

    I wouldn’t say that the Hitchhiker’s Guide it’s advertising anything. Looking at http://docs.python-guide.org/en/latest/dev/virtualenvs/

    It simply states, that virtualenv creates virtual environment wherever you create it, and virtualenvwrapper creates all of them in one place. Lists pros and cons, and it seems that it leaves the decision to ones consideration.

    • Paul Everitt Says:

      Under “Basic Usage”, it shows making a project directory, cd into it, and make the virtualenv there. You’re right though that it doesn’t advocate it per se.

  3. Tony Flury Says:

    I don’t think Pycharm insists on the virtualenv tree outside of the project, it is just the default directory in the Open dialog is your users home directory – it is possible to get PyCharm to put the virtualenv anywhere – and it does work (I do it with every new project). Having the virtualenv associated with the project makes complete sense to me.

    • Paul Everitt Says:

      I could have chosen a better phrase than “wants to”. But when doing the New Project wizard using a project template, if you make a new virtualenv in the project directory, then continue making the project, PyCharm warns you with a dialog that says “directory already exists”.

  4. stevepiercy Says:

    PyCharm does allow one to create a virtualenv inside the project directory, either as the root of the project directory or within. Here’s how to do it as the root:

    File > New Project…

    Select Project Type: Pure Python
    Location: ~/projects/projectname
    Interpreter: Create VirtualEnv (gear icon)

    Name: projectname
    Location: ~/projects/projectname
    Base interpreter: /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
    [ ] Inherit global site-packages (unchecked)

    $ tree -L 1 /Users/stevepiercy/projects/projectname
    /Users/stevepiercy/projects/projectname
    ├── bin
    ├── include
    └── lib

    ta-da!

    If you want the virtualenv to be at:

    ~/projects/projectname/env

    …you can adjust the values accordingly.

    • Paul Everitt Says:

      I wasn’t clear on this…it’s when you make a new project via a project template. You make a new virtual env, then continue making the project, and it warns that the directory already exists.

      • stevepiercy Says:

        I don’t get that in PyCharm 5.0.2. Maybe we’re talking about different procedures? In the New Project Wizard, do you choose a different Project Type than “Pure Python”?

  5. Paul Everitt Says:

    Yes, for example the Pyramid project template.

    • stevepiercy Says:

      OK, yeah, the wizard does not install Pyramid into the virtualenv, whether you first create a Pure Python project then attempt to create Pyramid project within that, or first create a Pyramid project. That’s a bug all right.

    • stevepiercy Says:

      As a workaround, create a Pure Python project, then install Pyramid into it. Now I can create a New Project with Pyramid as the Project Type. It’s not intuitive.

  6. Chris Arndt Says:

    There is no fixed 1:1 match between virtualenvs and projects in my book. Sometimes I have virtualenvs, in which I do “python setup.py develop” from several projects, because I’m testing bugs/features, which affect both projects. Sometimes I have several virtualenvs for the same project with different Python versions or different versions of dependencies. Yes, you should automate the latter use case with tox, which I usually do at some stage, but first I like do test out things interactively in different environments.Stupid me.

  7. Tin Tvrtković (@tintvrtkovic) Says:

    I usually create virtualenvs inside the project directory, in .venv. I used to name it just ‘venv’ (without the leading period) but then py.test would pick up tests from it. Another benefit is that the command to activate is the same for every project, so I can summon it easily in my shell.

    I’m happy with this arrangement, what’s the problem?

  8. Jeff H Says:

    +1 on inside. I too, like it all in one place and not scattered here and there.

  9. bbhaydon Says:

    I considered shifting to your method mostly because it dovetails with NodeJS/NPM convention of having the site/node packages in the project which seems neat for mixed js/python projects, but after rethinking it, it’s wrong, they’re wrong, and I don’t see how the docs validate your convention.

    A python project folder is self-contained in the sense that it’s just code that does nothing by itself. As source it’s decoupled from the environment. Therefore it’s transportable between environments and machines dependent on what versions and interpreters you support over the life of the project; 2.6, 3.5, pypy etc. Obviously you have version control but it isn’t necessary to give the project to someone.

    Your environment however isn’t transportable so you can’t just copy the project folder for someone or move to a different OS environment and expect it to just work unless you’re in a controlled lab. That should be reason enough for it not to be there. Instead you would give instructions in the README on how to install in the required packages and the pre-requisite python version(s) required.

    Pew and virtualenvwrapper make it trivial to have multiple environments for a single project (and also make environments potentially temporary), but even without them it would be straightforward to have projects/.venv/projectx as a sustainable convention to use for whatever the default python venv is for your project.

    As an aside, Nodejs NPM is leading to a new generation of devs making all the same packaging, environment mistakes that python with pip and pyvenv have solved. Fortunately using nodeenv and npm install -g for mixed python/js projects neatly slots into the python venv.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: