Category Archives: Python

Loading (and unloading) Python modules

I have a need to unload modules, or rather packages. It turns out to be a little messy, due to how import places stubs into sys.modules for lookups it knows will always fail. Or, rather, in order to properly unload modules, you have to know how modules were loaded in the first place.

Let’s say we have a package called package, i.e. a directory package/, with a __init__.py file inside it, along with the module files for package.

Let’s assume that __init__.py uses some other Python modules.

import os
import sys
... do stuff

What do you think will be in sys.modules after you import package?

You’ll see this

...
package <module 'package' from 'package\__init__.pyc'>
package.os None
package.sys None
...

The first entry is the code loaded for package. However, what are those next two entries? For example, the package.os entry records the fact that an import of os inside package is a redirection to a top-level os package, not a module inside package. This is an optimization for future imports, and nothing more. Furthermore, these exist only if PEP 328 is not in effect (explicit relative imports).

The basic drill for unloading a module is to remove any related entries from sys.modules, and then to hope you can remove all variables pointing inside the loaded module. Typically, this will be the symbol created from the import. So, all things being equal, you do this to import

import package

and this to “un-import”

del sys.modules["package"]
del package

and now the next import package that you do will cause Python to search for your package module and load it (import is a no-op on an already-loaded module).

You may or may not need to remove the stub entries. Presumably you are unloading and reloading in order to load a different version of the module (in my case, I’m bootstrapping to a newer version). If so, then it really depends on whether or not you overload names that might be in the search path. If you don’t ever do this, then don’t worry about it.

But it’s not really that simple. One of your modules might include others of your modules. For me, when I import package.cache (one of the modules inside package), I get this

package <module 'package' from 'package\__init__.pyc'>
package.__future__ None
package.cache <module 'package.cache' from 'package\cache.pyc'>
package.errno None
package.httplib None
package.lockfile <module 'package.lockfile' from 'package\lockfile.pyc'>
package.os None
package.package None
package.shutil None
package.socket None
package.stat None
package.sys None
package.tempfile None
package.threading None
package.time None
package.urllib None
package.util <module 'package.util' from 'package\util.pyc'>
package.zipfile None

My package.cache module in turn uses package.lockfile and package.util, and those import lots of standard modules, all of which get stubs. So to unload everything, I really need to remove everything starting with package. That might look something like this

for k in [x for x in sys.modules.keys() if x.startswith('package.')]:
  del sys.modules[k]
del sys.modules['package']
del package

which would remove everything inside package, and package, and then remove the local dictionary that contains package’s variables. My Python is probably not idiomatic enough yet, the use of that list comprehension seems strained.

os.link and os.symlink for Win32 Python

For some reason, the Win32 Python doesn’t implement hardlinks and symlinks, even though it’s there in the operating system as of Windows Vista. Here’s a simple version that’s a demonstration (a real version should memoize the lookup for the two functions, and probably handle Unicode, maybe some more error checking etc etc).

import os
import platform
import sys

source = sys.argv[1]
dest = sys.argv[2]
type = sys.argv[3]

def CreateHardLink(src, dst): 
  import ctypes
  flags = 1 if source is not None and os.path.isdir(src) else 0
  if not ctypes.windll.kernel32.CreateHardLinkA(dst, src, flags):
    raise OSError 

def CreateSymbolicLink(src, dst):
  import ctypes
  flags = 1 if source is not None and os.path.isdir(src) else 0
  if not ctypes.windll.kernel32.CreateSymbolicLinkA(dst, src, flags):
    raise OSError

if platform.system() == 'Windows':
  print 'hi there'
  os.link = CreateHardLink
  os.symlink = CreateSymbolicLink

if type == 'link':
  os.link(source, dest)
elif type == 'symlink':
  os.symlink(source, dest)
else:
  raise Exception('what?')

Windows + cmd.exe + Python + colored output

It’s not a make-or-break thing, but sometimes you want to make console output more readable.

http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python?rq=1

https://code.google.com/p/colorconsole/source/browse/colorconsole/win.py

And sometimes you want to know if you’re connected to a terminal or are redirecting to a file.

http://stackoverflow.com/questions/1077113/how-do-i-detect-whether-sys-stdout-is-attached-to-terminal-or-not

 

Deconstructing Deferred

This is a great thread about Twisted’s Deferred object, used to manage callbacks.

https://groups.google.com/forum/#!topic/python-tulip/ut4vTG-08k8/discussion

Patterns for clear and yet robust async programming are very important. This was a great write-up by Guido van Rossum explaining Deferred, and inferring its design decisions.

The actual Deferred documentation is here: http://twistedmatrix.com/documents/12.3.0/core/howto/defer.html