from python import confusion

[ Note: this article is about python3.3+ and does not talk about the differences between 2.7 and 3.3+ ]

At some point in your attempt to master Python, the import statement starts to cause trouble. In this article, I'm going to try to clarify how packages, modules and imports work together so you'll never have to play whack-a-mole again. I'll scaffold an 'import-naive' project structure and then dig into the problems it causes for imports and finally discuss ways to improve it.

Here's the layout:

  app/
    app.py                                         
    module_a.py                               
    lib/
      util.py                                      
      log.py
      format.py
    test/
      test_module_a.py                     
      test_util.py

And here it is again, annotated with the relevant import statements:

  app/
    app.py              # import module_a
    module_a.py         # from lib import util
    lib/
      util.py           # from . import log, format
      log.py
      format.py
    test/
      test_module_a.py  # import ..module_a
      test_util.py      # import ..lib.util

This works if you are in app/ and want to run the project via python app.py.

You can also run python module_a.py without issue.

However, you have an if __name__ == '__main__': ... block at the bottom to lib/util.py so you can run some module-specific code as a script. Unfortunately, running python lib/util.py as a script fails:

  Traceback (most recent call last):
    File "/path/to/app/lib/util.py", line 1, in <module>
      from . import log, format
      ^^^^^^^^^^^^^^^^^^^^^^^^^
  ImportError: attempted relative import with no known parent package

You might say to yourself: ah, my app needs to be a package. I'll go and add __init__.py files to the app directories because I've heard that will make an app into a package.

So you go and do that and now there is an __init__.py file in app/ and all of its subdirectories. Now it looks like this:

  app/
    __init__.py
    app.py              # import module_a
    module_a.py         # from lib import util
    lib/
      __init__.py
      util.py           # from . import log, format
      log.py
      format.py
    test/
      __init__.py
      test_module_a.py  # import ..module_a
      test_util.py      # import ..lib.util

When you run python lib/util.py, you get the same error. Then you cd into lib/ and run python util.py. But the error remains.

Let's read the error more closely: 'attempted relative import with no known parent package'. What does 'no parent package' mean? Isn't app the parent package of lib ?

It turns out that there is a global attribute __name__ available in every python module that can help us understand what's going on here.

What's in a __name__ ?

Every module has a global __name__ attribute whose value indicates how that module is being run. Is it being run directly as a script, or is it being imported? __name__ tells you that, and more.

Let's experiment by adding an lib/exp.py file and putting print(__name__) at the top.

  # file: lib/exp.py
  print(__name__)

If we run it directly with python lib/exp.py, it prints __main__.

If we import it into our top-level app.py file with from lib import exp and then run python app.py, lib/exp.py will still run that print statement, this time printing something interesting: lib.exp. The value in this case tells us the location of exp in the module heirarchy: exp is a submodule of the lib package.

This helps clarify why the imports fail when running python lib/util.py. In this context, it's the __main__ module and has no package context to use for module resolution.

And it will also probably clarify why the the lib/util.py imports succeed when lib/util.py is imported into app.py. As an imported module, lib is recognized as a package with a submodule called util. When util.py imports modules from ., python can resolve them against the registered package, lib.

It's possible to write Python for quite a while without ever needing to know this stuff explicitly. I encourage you to experiment on your own with the __name__ attribute and various test package structures to test your mental model of packages, modules and imports.

So, what exactly do we do about our python lib/util.py problem? We wanted to run the file directly...

Use python -m to run a module as a script

Assuming we're in the app/ directory, it turns out we can run python -m lib.util to tell python to run the lib/util.py module as a script. Note the new notation: package.submodule. We are no longer dealing with files but with packages and modules, which means that we're asking python to look through its sys.path for a package named lib and a submodule named lib.util.

Note that the location from which you invoke this command matters. It will look through directories in the sys.path to resolve it.

What is sys.path ?

sys.path is a list of file paths that are used in python's module resolution algorithm. The complexity of how this list is constructed at runtime is overwhelming. Fortunately, all you need to know to fix most straightforward user-defined module import issues is that python will put the parent directory of the __main__ script file at the beginning of sys.path, effectively trying to resolve imports against those first.

Let's print sys.path at the top of our lib/exp.py.

# file lib/exp.py

print(sys.path)
print(__name__)

If you run python lib/util.py, you will see /path/to/your/app/lib first in the list.

But if you run python -m lib.util, you will see that python adds the parent directory of the lib package to the sys.path, /path/to/your/app, which is how lib is seen as a package.

My Recommendation

If your app directory is relatively shallow and your app is not heavily modularized, I recommend using absolute imports for the sake of clarity and running your app via python -m app.app from the directory above the app root.

Below, I've adjusted the imports from the running example to their absolute format.

  # note: these imports require you to run the app via 'python -m app.app' from 
  # the directory that contains the root 'app' directory

  app/
    app.py              # import app.module_a
    module_a.py         # from app.lib import util
    lib/
      util.py           # from app.lib import log, format
      log.py
      format.py
    test/
      test_module_a.py  # import app.module_a
      test_util.py      # import app.lib.util

You might prefer to keep the relative imports. In which case, keep 'em! At least now you have a mental model of how import works off of sys.path and how to run your package/module as a script with python -m.

I hope you enjoyed this article. Consider referring to relative imports as 'package-relative imports' to dispel the impression that they are simply a shorthand convenience. New python programmers will thank you.