Python: Dynamically load all modules in a folder

In the __init__.py of the directory you want to include recursively:

    import pkgutil 

    __path__ = pkgutil.extend_path(__path__, __name__)
    for importer, modname, ispkg in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
          __import__(modname)

More Context

In my current project, I have a directory of data collectors that contain both a poller and a parser component. To keep things nice and clean, they’re located in a folder called collectors. The directory structure looks like this:

    collectors
    ├── base
    │   ├── __init__.py
    │   ├── parser.py
    │   ├── poller.py
    ├── graphite
    │   ├── __init__.py
    │   ├── parser.py
    │   ├── poller.py
    ├── analytics
    │   ├── __init__.py
    │   ├── parser.py
    │   ├── poller.py
    └── __init__.py

base.poller and base.parser are factory classes, which are responsible for creating instances of either graphite.* or analytics.*. This means that these are the only classes I need to expose. To do this, I use __all__ in __init__.py in the collectors folder:

    from base.poller import Poller
    from base.parser import Parser

    __all__ = ["Poller", "Parser"]

This means that I can use the following in my code:

    from collectors import Poller, Parser

The next thing to do, is import all of the graphite and analytics modules, as they register themselves with the factories on load. I don’t need to access them directly in my code as I access things via the factories. To do this, I use the following code (modified based on code from nulledge).

    import pkgutil 

    __path__ = pkgutil.extend_path(__path__, __name__)
    for importer, modname, ispkg in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
          __import__(modname)

Then in my code, I use the following to access the modules that registered themselves:

    gpoller = Poller.create("graphite")

Michael is a polyglot software engineer, committed to reducing complexity in systems and making them more predictable. Working with a variety of languages and tools, he shares his technical expertise to audiences all around the world at user groups and conferences. You can follow @mheap on Twitter

Thoughts on this post

Aurelien 2014-11-19

Nice article. Here is an alternative without __init__.py (or any other extra file): https://gitlab.com/aurelien-lourot/importdir
It allows you to import all modules in a folder with only two lines.

freemansionn 2018-07-30

Didn’t know how to organize project structure clean and keep references to my model classes safe. This article helped me. Thanks Michael!

Leave a comment?

Leave a Reply