Python: Dynamically load all modules in a folder

27 Jan 2014 in TIL

In the __init__.py of the directory you want to include recursively:

python
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for importer, modname, ispkg in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
__import__(modname)

More Context

In my current project, I have a directory of data collectors that contain both a poller and a parser component. To keep things nice and clean, they're located in a folder called collectors. The directory structure looks like this:

bash
collectors
├── base
│   ├── __init__.py
│   ├── parser.py
│   ├── poller.py
├── graphite
│   ├── __init__.py
│   ├── parser.py
│   ├── poller.py
├── analytics
│   ├── __init__.py
│   ├── parser.py
│   ├── poller.py
└── __init__.py

base.poller and base.parser are factory classes, which are responsible for creating instances of either graphite.* or analytics.*. This means that these are the only classes I need to expose. To do this, I use __all__ in __init__.py in the collectors folder:

python
from base.poller import Poller
from base.parser import Parser
__all__ = ["Poller", "Parser"]

This means that I can use the following in my code:

python
from collectors import Poller, Parser

The next thing to do, is import all of the graphite and analytics modules, as they register themselves with the factories on load. I don't need to access them directly in my code as I access things via the factories. To do this, I use the following code (modified based on code from nulledge).

python
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for importer, modname, ispkg in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
__import__(modname)

Then in my code, I use the following to access the modules that registered themselves:

python
gpoller = Poller.create("graphite")