Posted on Aug 31, 2020 • Updated on Nov 10, 2020
UPDATE - the updated version of this project is here - //github.com/unfor19/python-project/wiki
By the end of this blog post, you will be able to:
- Create a well-structured Python project
- Use relative and absolute imports in a Python project
- Invoke a specific module from the command-line [terminal]
Link to GitHub repository - unfor19/python-project
python-project
Python project structure, relative imports, absolute imports, packages, and modules. Let's make it simpler.
Getting Started
Executing modules from the project's root directory [top-level package]
- main.py
meirgabay@~/python-project [master]$ python main.py
My Path: python-project/main.py
Insert your name: willy
Hello willy
meirgabay@~/python-project [master]$ python -m appy
My Path: python-project/appy/__main__.py
Insert your name: willy
Hello willy
Enter fullscreen mode Exit fullscreen mode
- appy/core/app.py
# Contains relative imports - `..utils.message`
meirgabay@~/python-project [master]$ python appy/core/app.py
Traceback [most recent call last]:
File "appy/core/app.py", line 1, in
from ..utils import message
ImportError: attempted relative import with no known parent package
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
Enter fullscreen mode Exit fullscreen mode
- appy/utils/message.py
# Doesn't contain relative imports, so no exceptions were raised
meirgabay@~/python-project [master]$ python appy/utils/message.py
My Path: python-project/appy/utils/message.py
meirgabay@~/python-project [master]$ python -m appy.utils.message
My Path: python-project/appy/utils/message.py
Enter fullscreen mode Exit fullscreen mode
Questions and Answers [Q&A]
Project, Packages, Modules and Scripts, what are they?
- Project - a directory, also known as the top-level package, which contains packages and modules
- Package [in a project] - a directory which contains modules and/or packages [sub-directories]
- Script - a Python script [
Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
- which can be exected from the terminal
- Module - a Python script [
Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
- which can be imported with
Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
5 and
6Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
What about Packages which are not part of a project?
- Package [built-in] - a package which is shipped with Python and can be imported with
Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
5 and
6Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
- Package [pip] - a package which is installed with pip and can be imported with
Contains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
5 and
6. Think about it, pip stands for Package Installer for PythonContains relative imports -
ImportError: attempted relative import with no known parent package meirgabay@/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy `File "appy/core/app.py", line 1, in from ..utils import message
How do I import Packages and Modules that I've created?
- Python project's packages and modules can be imported with relative paths from any module which is part of the same project. An example is available in
- If you intend to import a package or a module which is not part of the same project, you'll have to use absolute paths. This can be done with importlib, see this StackOverflow answer.
Do I need the __init__.py file?
- Short answer - no
- In previous versions of Python, you had to create the
Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
1 file in each directory that you want to import as a package, they were called regular packages. From version 3.3+ it is not required anymore - Implicit Namespace Packages, packages without an
1 file are called namespace packages.Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
Why do relative imports raise a problem in pylint?
The error -
# Doesn't contain relative imports, so no exceptions were raised
meirgabay@~/python-project [master]$ python appy/utils/message.py
My Path: python-project/appy/utils/message.py
meirgabay@~/python-project [master]$ python -m appy.utils.message
My Path: python-project/appy/utils/message.py
3
- Short answer - I don't know
- All I can say is that it doesn't happen with flake8
Is it possible to invoke a function from the terminal?
- Short answer - it depends
- Trying to invoke a function from the terminal, such as
4, will raise the ModuleNotFound exception. A package must be imported before invoking one of its functions.Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
meirgabay@~/python-project [master]$ python -m appy.core.app.main
/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: __path__ attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
Enter fullscreen mode Exit fullscreen mode
- Since you can't invoke
Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
5 directly from the terminal, calling it from the
Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
6 block enables executing it from the terminal. It's possible to pass arguments, but it's a bit ugly, read the docs to learn how. The following example attempts to execute the module
Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
7, which in turn call its
6 blockDoesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
Enter fullscreen mode Exit fullscreen mode
- If the
Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
9 is a subdirectory of the project, such as
meirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
0, an attempt to execute a module which contains relative imports, will raise the exception below. Remember, your
Doesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
9 should always be the project's root directory, in this case it's
2.meirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
# PWD is `appy`
meirgabay@~/python-project/appy [master]$ python -m core.app
Traceback [most recent call last]:
File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
return _run_code[code, main_globals, None,
File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
exec[code, run_globals]
File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in
from ..utils import message
ValueError: attempted relative import beyond top-level package
Enter fullscreen mode Exit fullscreen mode
- It doesn't happen when invoking
meirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
3, since
3 doesn't use relative importsmeirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
meirgabay@~/python-project/appy [master]$ python utils/message.py
My Path: python-project/appy/utils/message.py
meirgabay@~/python-project/appy [master]$ python -m utils.message
My Path: python-project/appy/utils/message.py
Enter fullscreen mode Exit fullscreen mode
- Invoking a function from the terminal is also possible by using the
meirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
5 flag. Surprise, it's possible to pass arguments in a more intuitive way, for example
6meirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
meirgabay@~/python-project [master]$ python -c "import appy.core.app as app; app.main[]"
Insert your name: willy
Hello willy
Enter fullscreen mode Exit fullscreen mode
What are the available command-line flags in Python?
- Read the docs - using cmdline
- In this tutorial, we used both
meirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
5 and
8 flagsmeirgabay@~/python-project [master]$ python -m appy.core.app.main /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: path attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
Why is it possible to execute
meirgabay@~/python-project [master]$ python -m appy.core.app.main
/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' [ModuleNotFoundError: __path__ attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main']
9?
The appy/__main__.py file acts like the
# Doesn't contain relative imports, so no exceptions were raised
meirgabay@~/python-project [master]$ python appy/utils/message.py
My Path: python-project/appy/utils/message.py
meirgabay@~/python-project [master]$ python -m appy.utils.message
My Path: python-project/appy/utils/message.py
6 code snippet, but on packages. This enables the
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
1 package to be executed with
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
2 or with runpy
meirgabay@~/python-project [master]$ python -m appy
My Path: python-project/appy/__main__.py
Insert your name: willy
Hello willy
Enter fullscreen mode Exit fullscreen mode
What's
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
3 and why do you use it in
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
4?
The
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
3 package provides the ability to run modules from a module [Python script].
main.py
import runpy
def main[]:
# import a package, and pass all current global variables to it
appy_package = runpy.run_module[mod_name="appy", init_globals=globals[]]
# import the function script_path[] from the submodule message, and execute it
appy_package['message'].script_path[__file__]
# execute the function main[], which is located in appy/__main__.py
appy_package['main'][]
if __name__ == "__main__":
main[]
Enter fullscreen mode Exit fullscreen mode
What's
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
6?
The official definition from the
Return a dictionary representing the current global symbol table. This is always the dictionary of the current module [inside a function or method, this is the module where it is defined, not the module from which it is called].
Example - Expand/Collapse
# Contains relative imports - `..utils.message`
meirgabay@~/python-project [master]$ python appy/core/app.py
Traceback [most recent call last]:
File "appy/core/app.py", line 1, in
from ..utils import message
ImportError: attempted relative import with no known parent package
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
0
Enter fullscreen mode Exit fullscreen mode
Why do you have a weird path with
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
7 when you run Python?
In some of the examples you might have seen that my Python binary is located in
# Contains relative imports - `..utils.message`
meirgabay@~/python-project [master]$ python appy/core/app.py
Traceback [most recent call last]:
File "appy/core/app.py", line 1, in
from ..utils import message
ImportError: attempted relative import with no known parent package
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
1
Enter fullscreen mode Exit fullscreen mode
This is because I'm using pyenv, the official definition from the docs
pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.
- pyenv great for checking backwards compatibility
- Switching to a different version
- Install relevant version -
8meirgabay@~/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy
- Run
9meirgabay@~/python-project [master]$ python -m appy.core.app My Path: python-project/appy/core/app.py Insert your name: willy Hello willy
- For a day to day use
- Install relevant version -
0PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
- Add
PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
1 to your terminal's
PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
2 or
PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
3 [
PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
4,
PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
5,
6]PWD is `appy
ValueError: attempted relative import beyond top-level package `File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code[code, main_globals, None, File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code exec[code, run_globals] File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in from ..utils import message
Examples - Expand/Collapse
# Contains relative imports - `..utils.message`
meirgabay@~/python-project [master]$ python appy/core/app.py
Traceback [most recent call last]:
File "appy/core/app.py", line 1, in
from ..utils import message
ImportError: attempted relative import with no known parent package
meirgabay@~/python-project [master]$ python -m appy.core.app
My Path: python-project/appy/core/app.py
Insert your name: willy
Hello willy
2
Enter fullscreen mode Exit fullscreen mode
Is there a good framework for creating a Python CLI?
- Short answer - yes! more than a few!
- The Click Framework is amazing! I used it in githubsecrets and frigga NOTE: you'll discover that I have
1 files in my projects, that's a nasty old habit, I promise I will drop them, I promiseDoesn't contain relative imports, so no exceptions were raised
meirgabay@/python-project [master]$ python appy/utils/message.py My Path: python-project/appy/utils/message.py meirgabay@/python-project [master]$ python -m appy.utils.messageMy Path: python-project/appy/utils/message.py
- Read more about the available Python CLI frameworks in this great blog post - Building Beautiful Command Line Interfaces with Python
Where can I find a well-structured Python project?
- sampleproject - the official Python sample project
- Django - a complex Python project
- requests - a non-complex Python project
Final words
I hope that this blog post helped you in understanding how to create a well-structured Python project, and if it did, then Heart, Clap, Star and share!
How to import relative path package in Python?
To import a module from a relative path in Python, you can use the importlib. import_module[] function from the importlib module. This function allows you to import a module by specifying its name and the relative path to the module.
What is the top level module in Python?
“Top-level code” is the first user-specified Python module that starts running. It's “top-level” because it imports all other modules that the program needs. Sometimes “top-level code” is called an entry point to the application.
Why are Python imports so hard?
Python's import system is powerful, but also quite complicated. Until the release of Python 3.3, there was no comprehensive explanation of the expected import semantics, and even following the release of 3.3, the details of how sys. path is initialised are still somewhat challenging to figure out.
What is the difference between relative and absolute imports in Python?
Absolute vs Relative Imports Absolute is more explicit on where the module exists in a project hierarchy, relative could be used if you don't want to type long imports and don't need to know the position of the module relative to the modules it is importing.