PyInstaller – Separating the executable from the ‘onedir’

Having had a quick play with py2exe, cx_Freeze and PyInstaller the one that worked for me least problematically was PyInstaller. I'm impressed with its ability to do the right thing out-of-the-box for a non-trivial app that uses PySide, PyOpenGL and NumPy among other 3rd-party libraries. My one real bugbear is that I don't really like either its 'onedir' or its 'onefile' modes, and there aren't any other options. As the name implies, 'onefile' bundles the whole mess into a single executable, which is sort of appealing, but this means that when the executable is run all those bundled files and directories have to be extracted somewhere, which adds a delay (which doesn't seem to be too bad on my developer box, but still). Furthermore, according to the manual, on Windows "… the files can't be cleaned up afterwards because Python does not call FreeLibrary", which means you have a bunch of stuff sitting in a temp directory (e.g. 'C:\\Users\\Me\\AppData\\Local\\Temp\\_MEI188722') with no real control over when it will be cleaned up (if ever). This kind of sets off my OCD, and I'd much rather have it be clear to both me and the end-user where all the files and directories relating to my application are (explicit is better than implicit). Which brings us to 'onedir' mode, which as the name implies shoves everything inside one directory, with the executable sitting alongside all the crap it depends on. This doesn't have to unpack anything from the executable when run, and there's no messing about with obscure temp directories, but the problem is that the 'one directory' just looks like a complete mess.
Again, this is probably just my OCD playing up, but I want to present a clean directory to my end users. Instead of this:

I want something more like this:

where 'lib' contains all the files and directories from the first screenshot that my app depends on. Unfortunately, there isn’t an easy way to get PyInstaller to do this. You can get most of the way there with only a little messing about though.

So the general idea is that we have a ‘lib’ directory (or whatever you want to call it) that contains the dependencies but that doesn’t contain the executable. If you just create one and move all the dependencies into it you’ll get errors at runtime because the dependencies can’t be found – nothing knows about your new lib directory. So the main thing you need to do is add your lib directory to the sys.path. You can’t do this in your main application module though because by the time that is run it is too late – the bootstrapping code will import stuff before your main script is run. So the way to do it is with a runtime hook. This is just a Python module; call it whatever you want and put something like the following in it:

import sys
import os
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "lib"))

To have PyInstaller use this you can just pass the path to it as an argument to pyinstaller (alternatively you can specify it in your .spec file):

pyinstaller --onedir --windowed --runtime-hook=".\_pyinstaller_hooks\runtime_hooks\use_lib.py" …

Happily, this gets us most of the way there. Playing with this, by manually moving the built files about, I find that I can move everything into the lib directory except python27.dll (which makes sense), the ‘eggs’ directory and the PySide .pyd files. If you’re not using PySide then that last obviously won’t be an issue, so with a bit of luck just by amending the path in the runtime hook you should be able to make your ‘one dir’ distribution look like this:

which I’m sure you’ll agree is a lot nicer than the default. You’ll need to experiment to see whether the dependencies in your application can tolerate being moved into a separate directory from the executable; as noted PySide doesn’t like it (and I can’t find an easy way to fix this) but other things like NumPy seem to be perfectly happy. In case it’s not obvious, since PyInstaller isn’t set up to build this way out of the box you’ll need to move the files yourself – I just use a batch file that firstly calls pyinstaller to build the ‘one dir’ and then creates the lib directory and moves the relevant stuff into it. Finally, I’ve only tried this on Windows.

Update 15th August 2014

Here's the contents of my .spec file, as requested by Rishi Sharma:

# -*- mode: python -*-
a = Analysis(['..\\src\\my_app.py'],
             pathex=['C:\\dev\\python_projects\\my_app\\dist'],
             hiddenimports=[],
             hookspath=['.\\_pyinstaller_hooks\\additional_hooks'],
             runtime_hooks=['.\\_pyinstaller_hooks\\runtime_hooks\\use_lib.py'])
pyz = PYZ(a.pure)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='my_app.exe',
          debug=False,
          strip=None,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=None,
               upx=True,
               name='my_app')

Note that this is just what pyinstaller automatically generates when I call it with the arguments shown above - I haven't tweaked it. The part highlighted shows how to set the hook and runtime hook paths - obviously you'll need to adjust it for your own paths. Hope this helps.