Python Program Distribution and Installation Methods

🐍 tldr; There are many methods to distribute and install Python programs. Here are some of the ways I've tinkered with recently. Recommendation: in most cases, use pipx. 🔧

Introduction

I like Python. I’m not sure why…maybe because it seems the closest programming language to English, or maybe because it’s so flexible that you can be messy with it. A lot of people dislike it because it’s not typed, and because you can be messy with it–one can write some pretty unstructured Python code. Also, it doesn’t compile into a single binary. I think that is part of the reason why Go has gained a lot of traction, and now, in a similar vein, we have Rust too. Another problem is that you have to deal with Python versions on the machine where the application is installed. It might be nice to distribute the Python runtime with the application.

So, problems to solve with distributing Python programs, at least by default:

1. You don’t have a single binary….instead you have an entrypoint into a bunch of Python code and dependencies
2. You don’t have a single Python version embedded…you have to deal with all kinds of different system versions
3. You typically have a lot of dependencies, Python has a lot of great libraries to take advantage of
4. Some dependencies need to be compiled, netifaces as an example, so sometimes you might need a massive toolchain to build them, which makes the user experience a bit more complex (though Python Wheels help with this)

This led me to try a few different ways to distribute Python programs that might help with these problems. So far there have been five or so ways I’ve tinkered with to distribute Python programs.

1. AppImage - NOTE: Didn’t get this working
2. PyInstaller - This did work and created a nice binary
3. Pip
4. Installer script
5. Pipx - Probably the best choice for most situations

Installation and Distribution Methods

AppImage

AppImage is a very interesting idea, but I couldn’t get it working with my Python application.

From the AppImage website:

“As a user, I want to download an application from the original author, and run it on my Linux desktop system just like I would do with a Windows or Mac application.”

“As an application author, I want to provide packages for Linux desktop systems, without the need to get it ‘into’ a distribution and without having to build for gazillions of different distributions.”

I tinkered with AppImage for a while, but ended up just using PyInstaller. AppImage can package anything, whereas something like PyInstaller is more specific to Python applications. I think this is what tripped me up.

I’ll come back to AppImage later as I think it’s a great idea, and I have applications I use which are distributed as AppImages–just download, mark as executable and run, all in one file.

PyInstaller

PyInstaller bundles a Python application and all its dependencies into a single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 3.8 and newer, and correctly bundles many major Python packages such as numpy, matplotlib, PyQt, wxPython, and others. - PyInstaller

There are a few ways to get a binary out of a Python program. PyInstaller is one of them, and this is the method that seems to work best for me.

Here I build a nice single binary that I can distribute.

pyinstaller \
            --clean \
            --onefile \
            --name coolprog \
            --workpath pyinstaller-build \
            --distpath dist \
            --hidden-import yaml \
            --hidden-import netifaces \
            --hidden-import redis \
            --hidden-import coolprog.cli \
            src/coolprog_main.py

This created a nice relatively small binary.

du -hsc dist/coolprog
16M	dist/coolprog
16M	total

All you have to do is distribute the binary! Having a single binary that works, that contains the Python runtime is amazing and feels like magic.

One issue that I have is that apparently sometimes you have to give it some hints regarding the dependencies, e.g. --hidden-import. I’m not clear on how you would know what hints to give.

Benefits:

• Everything in one binary
• This includes the Python runtime as well!
• And all the application dependencies

Pip

Pip is the Python package manager. It’s what you typically use to install Python packages. If you’ve used Python, you’ve probably used pip.

A few points:

• pip installs your application and dependencies from pypi.org
• Anyone (yes anyone) can sign up for a PyPI account and upload their own packages (which is amazing in itself!)
• There is also a test PyPI server, which is useful for testing

Once you’ve signed up for a PyPI account, setup your application, you can simply upload it to PyPI with something like twine, and then once it is uploaded, you can install it with pip.

pip install coolprog

However, in modern Linux distributions, you will get this kind of message:

ubuntu@coolprog-test:~$ pip install coolprog
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.

With this in mind, you will likely want to use pipx instead of pip as you don’t want to mess with the system Python, say for example by breaking it during the installation of your application. That would not make you unpopular and would provide a poor user experience.

Pipx

Pipx is a tool for installing and running Python applications in isolated environments. It’s like pip but specifically for Python applications rather than libraries.

The key benefits of pipx are:

1. It installs each application in its own isolated virtual environment, preventing dependency conflicts between different applications
2. It makes Python applications available globally on your system while keeping their dependencies contained
3. It’s easy to install, upgrade, and uninstall applications without affecting other Python tools

Here is an example of installing a Python application with pipx:

pipx install coolprog

e.g. output:

ubuntu@coolprog-test:~$ pipx install coolprog
  installed package coolprog 0.2.0, installed using Python 3.12.3
  These apps are now globally available
    - coolprog
⚠️  Note: '/home/ubuntu/.local/bin' is not on your PATH environment variable. These apps will
    not be globally accessible until your PATH is updated. Run `pipx ensurepath` to
    automatically add it, or manually modify your PATH in your shell's config file (i.e.
    ~/.bashrc).
done! ✨ 🌟 ✨

So, just add it to your path and you’re good to go.

ubuntu@coolprog-test:~$ pipx ensurepath
Success! Added /home/ubuntu/.local/bin to the PATH environment variable.

Consider adding shell completions for pipx. Run 'pipx completions' for instructions.

You will need to open a new terminal or re-login for the PATH changes to take effect.

Otherwise pipx is ready to go! ✨ 🌟 ✨

Nice!

Installer Script

Working with installer scripts was my first approach before moving to PyInstaller or pipx. The classic curl | bash installation method, while discouraged for security reasons, offers simplicity that some people appreciate. The challenge lies in creating an installer script that provides both convenience and a great user experience. You need to consider how the installation impacts the user’s system–what gets installed where, and how it interacts with existing components, etc., etc.

In the script I eventually settled on installing the application in its own Python virtual environment, which ironically ended up being quite similar to pipx’s approach. This makes sense, as isolated environments help avoid dependency conflicts while maintaining a clean, manageable installation. Sure you have a bit more complexity in the number of virtual environments you have to manage, but if there is an abstraction managing it, e.g. pipx, then it’s doable.

In the end, an installer script might make sense, but if pipx will work, it may be best to simply use pipx.

Other Methods

I didn’t try any of these, but some other methods I’ve seen:

• Cx_Freeze
• Flatpak
• Py2Exe
• bbfreeze
• py2app
• PyOxidizer???
• Nuitka???
• Docker

And probably many more, some of which are new, and some of which are getting long in the tooth.

Conclusion

Distributing Python applications isn’t easy. You are installing something into someone’s computer, or server, and you have to make sure it works, that it isn’t going to break their system or their Python environment, and that it is as easy as possible to install. But doing all that is more difficult than it sounds. Though, having now discovered pipx, I think it is the best choice for most situations.

I find all the work outside of writing code for an application to be much, much more complex than writing the code itself, and I’ve written about it here.

🤔 Related: Check out my post about the pain points in programming beyond just writing code. 😅