demandimport is broken on py3k

Renato Cunha renatoc at gmail.com
Fri Aug 6 22:16:01 CDT 2010


Hello!

The problem
===========

Maybe it took me more than it should to realize this, but demandimport is
"broken" on py3k.

Even though a test like

    $ echo 'print("Hello, world!")' > foo.py
    $ python3
    Python 3.1.2 (r312:79147, Apr  1 2010, 09:12:21) 
    [GCC 4.4.3 20100316 (prerelease)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from mercurial import demandimport; demandimport.enable()
    >>> import foo
    >>> foo.a
    Hello, world!
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "mercurial/demandimport.py", line 79, in __getattribute__
        return getattr(self._module, attr)
    AttributeError: 'module' object has no attribute 'a'

Will succeed, I have evidence that it isn't effectively working when called
from within hg: some import cycles fail badly (yes, there are a few of them).
To support this evidence, I put a "print('Hello world')" inside help.py, run
hg and got the following:

    $ python3 hg
    Hello world
    Traceback (most recent call last):
      File "hg", line 23, in <module>
        import mercurial.dispatch
      File "~/mercurial/hg2to3/mercurial/dispatch.py", line 10, in <module>
        from . import util, commands, hg, fancyopts, extensions, hook, error
      File "~/mercurial/hg2to3/mercurial/commands.py", line 12, in <module>
        from . import hg, util, revlog, bundlerepo, extensions, copies, error
      File "~/mercurial/hg2to3/mercurial/hg.py", line 11, in <module>
        from . import localrepo, bundlerepo, httprepo, sshrepo, statichttprepo
      File "~/mercurial/hg2to3/mercurial/localrepo.py", line 13, in <module>
        from . import util, extensions, hook, error
      File "~/mercurial/hg2to3/mercurial/extensions.py", line 11, in <module>
        from .help import moduledoc
      File "~/mercurial/hg2to3/mercurial/help.py", line 12, in <module>
        from . import extensions
    ImportError: cannot import name extensions

An equivalent invocation with python2 gives no Exception, nor a "Hello world"
string.

I investigated a bit and found the culprit in demandimport.py:

    if level is not None:
        # from . import b,c,d or from .a import b,c,d
        return _origimport(name, globals, locals, fromlist, level)

(http://selenic.com/repo/hg/file/05deba16c5d5/mercurial/demandimport.py#l102)

The problem here is that _demandimport assumes that, by default, the python
interpreter won't pass a value to __import__'s level argument (_demandimport
implements the __import__ API: __import__(name, globals={}, locals={},
fromlist=[], level=-1) -> module). Even though that seems to be true in python
2.x, that's not the case in py3k. Quoting the python documentation, "level
specifies whether to use absolute or relative imports." To make things worse,
as all imports in py3k are absolute, level's default value in py3k is 0. (Zero
means "only perform absolute import" in __import__.)

[1] http://docs.python.org/py3k/library/functions.html#__import__

One can check the default arguments passed to __import__ with the following
test:

1) Suppose you have the following script:

def i(name, globals = None, locals = None, fromlist = None, level = None):
    print("level = %s" % level)
try:
    import __builtin__
    __builtin__.__import__ = i
except:
    import builtins
    builtins.__import__ = i

import bar

2) Call it in python2:

    $ python2 foo.py
    level = None

3) Call it in python3:

    $ python3 foo.py
    level = 0

4) Oops?

Solutions
=========

At first, I thought that adding a check to see if level was neither None nor
zero would do it. But then I remembered that in the py3k port, mercurial's
modules are imported in the format "from . import module", which means "level"
is at least 1. Playing a bit more with the script above has shown that an
import with the format "from . import foo" has the following (relevant) arguments:

    level = 1
    fromlist: ('foo',)
    name = , len(name) = 0

So, correct me if I'm wrong, but I believe that a check equivalent to the
following would be needed to correctly handle the python 2.x and python 3.x
cases:

diff --git a/mercurial/demandimport.py b/mercurial/demandimport.py
--- a/mercurial/demandimport.py
+++ b/mercurial/demandimport.py
@@ -99,7 +99,9 @@
                 return locals[base]
         return _demandmod(name, globals, locals)
     else:
-        if level is not None:
+        if level is not None and \
+           level != 0 and \
+           not (level == 1 and len(name) == 0):
             # from . import b,c,d or from .a import b,c,d
             return _origimport(name, globals, locals, fromlist, level)
         # from a import b,c,d

Any thoughts on that? Too hacky? Am I missing something. I probably _am_
missing something, as I've just focused on the obvious problem for now...

Regards,
-- 
Renato Cunha <http://renatocunha.com>
Blog: http://valedotrovao.com
"Perhaps ignorance is bliss".
            -- Sheldon Cooper


More information about the Mercurial-devel mailing list