mercurial-scm/hg: contrib/import-checker.py annotate

annotate contrib/import-checker.py @ 25701:1f88c0f6ff5a

import-checker: resolve relative imports "from . import X" will produce an ImportFrom ast node with .module = None. This resulted in a run-time error from attempting to concatenate None with a str. Another problem with relative imports is that the prefix may be dynamic based on the "level" attribute of the import. e.g. "from ." has level 1 and "from .." has level 2. We teach the "fromlocal" function how to cope with relative imports. Where appropriate, the consumer passes in the level so relative module names may be resolved properly.

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Sun, 28 Jun 2015 09:36:58 -0700
parents	328739ea70c3
children	ab2c5163900e

rev	line source
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	1 import ast
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	2 import os
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	3 import sys
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	4
20198 f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	5 # Import a minimal set of stdlib modules needed for list_stdlib_modules()
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	6 # to work when run from a virtualenv. The modules were chosen empirically
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	7 # so that the return value matches the return value without virtualenv.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	8 import BaseHTTPServer
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	9 import zlib
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	10
22975 461342e1c8aa import-checker: check modules for pure Python build correctly FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22974 diff changeset	11 def dotted_name_of_path(path, trimpure=False):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	12 """Given a relative path to a source file, return its dotted module name.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	13
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	14 >>> dotted_name_of_path('mercurial/error.py')
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	15 'mercurial.error'
22975 461342e1c8aa import-checker: check modules for pure Python build correctly FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22974 diff changeset	16 >>> dotted_name_of_path('mercurial/pure/parsers.py', trimpure=True)
461342e1c8aa import-checker: check modules for pure Python build correctly FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22974 diff changeset	17 'mercurial.parsers'
20383 4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	18 >>> dotted_name_of_path('zlibmodule.so')
4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	19 'zlib'
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	20 """
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	21 parts = path.split('/')
20391 466e4c574db0 import-checker: handle standard modules with arch in the filename Mads Kiilerich <madski@unity3d.com> parents: 20386 diff changeset	22 parts[-1] = parts[-1].split('.', 1)[0] # remove .py and .so and .ARCH.so
20383 4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	23 if parts[-1].endswith('module'):
4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	24 parts[-1] = parts[-1][:-6]
22975 461342e1c8aa import-checker: check modules for pure Python build correctly FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22974 diff changeset	25 if trimpure:
461342e1c8aa import-checker: check modules for pure Python build correctly FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22974 diff changeset	26 return '.'.join(p for p in parts if p != 'pure')
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	27 return '.'.join(parts)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	28
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	29 def fromlocalfunc(modulename, localmods):
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	30 """Get a function to examine which locally defined module the
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	31 target source imports via a specified name.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	32
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	33 `modulename` is an `dotted_name_of_path()`-ed source file path,
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	34 which may have `.__init__` at the end of it, of the target source.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	35
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	36 `localmods` is a dict (or set), of which key is an absolute
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	37 `dotted_name_of_path()`-ed source file path of locally defined (=
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	38 Mercurial specific) modules.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	39
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	40 This function assumes that module names not existing in
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	41 `localmods` are ones of Python standard libarary.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	42
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	43 This function returns the function, which takes `name` argument,
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	44 and returns `(absname, dottedpath, hassubmod)` tuple if `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	45 matches against locally defined module. Otherwise, it returns
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	46 False.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	47
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	48 It is assumed that `name` doesn't have `.__init__`.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	49
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	50 `absname` is an absolute module name of specified `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	51 (e.g. "hgext.convert"). This can be used to compose prefix for sub
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	52 modules or so.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	53
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	54 `dottedpath` is a `dotted_name_of_path()`-ed source file path
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	55 (e.g. "hgext.convert.__init__") of `name`. This is used to look
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	56 module up in `localmods` again.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	57
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	58 `hassubmod` is whether it may have sub modules under it (for
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	59 convenient, even though this is also equivalent to "absname !=
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	60 dottednpath")
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	61
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	62 >>> localmods = {'foo.__init__': True, 'foo.foo1': True,
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	63 ... 'foo.bar.__init__': True, 'foo.bar.bar1': True,
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	64 ... 'baz.__init__': True, 'baz.baz1': True }
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	65 >>> fromlocal = fromlocalfunc('foo.xxx', localmods)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	66 >>> # relative
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	67 >>> fromlocal('foo1')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	68 ('foo.foo1', 'foo.foo1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	69 >>> fromlocal('bar')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	70 ('foo.bar', 'foo.bar.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	71 >>> fromlocal('bar.bar1')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	72 ('foo.bar.bar1', 'foo.bar.bar1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	73 >>> # absolute
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	74 >>> fromlocal('baz')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	75 ('baz', 'baz.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	76 >>> fromlocal('baz.baz1')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	77 ('baz.baz1', 'baz.baz1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	78 >>> # unknown = maybe standard library
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	79 >>> fromlocal('os')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	80 False
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	81 >>> fromlocal(None, 1)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	82 ('foo', 'foo.__init__', True)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	83 >>> fromlocal2 = fromlocalfunc('foo.xxx.yyy', localmods)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	84 >>> fromlocal2(None, 2)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	85 ('foo', 'foo.__init__', True)
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	86 """
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	87 prefix = '.'.join(modulename.split('.')[:-1])
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	88 if prefix:
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	89 prefix += '.'
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	90 def fromlocal(name, level=0):
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	91 # name is None when relative imports are used.
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	92 if name is None:
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	93 # If relative imports are used, level must not be absolute.
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	94 assert level > 0
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	95 candidates = ['.'.join(modulename.split('.')[:-level])]
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	96 else:
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	97 # Check relative name first.
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	98 candidates = [prefix + name, name]
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	99
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	100 for n in candidates:
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	101 if n in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	102 return (n, n, False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	103 dottedpath = n + '.__init__'
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	104 if dottedpath in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	105 return (n, dottedpath, True)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	106 return False
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	107 return fromlocal
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	108
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	109 def list_stdlib_modules():
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	110 """List the modules present in the stdlib.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	111
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	112 >>> mods = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	113 >>> 'BaseHTTPServer' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	114 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	115
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	116 os.path isn't really a module, so it's missing:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	117
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	118 >>> 'os.path' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	119 False
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	120
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	121 sys requires special treatment, because it's baked into the
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	122 interpreter, but it should still appear:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	123
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	124 >>> 'sys' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	125 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	126
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	127 >>> 'collections' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	128 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	129
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	130 >>> 'cStringIO' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	131 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	132 """
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	133 for m in sys.builtin_module_names:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	134 yield m
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	135 # These modules only exist on windows, but we should always
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	136 # consider them stdlib.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	137 for m in ['msvcrt', '_winreg']:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	138 yield m
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	139 # These get missed too
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	140 for m in 'ctypes', 'email':
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	141 yield m
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	142 yield 'builtins' # python3 only
24669 fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules Matt Harbison <matt_harbison@yahoo.com> parents: 24668 diff changeset	143 for m in 'fcntl', 'grp', 'pwd', 'termios': # Unix only
fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules Matt Harbison <matt_harbison@yahoo.com> parents: 24668 diff changeset	144 yield m
20197 761f2929a6ad import-checker: refactor sys.path prefix check (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20038 diff changeset	145 stdlib_prefixes = set([sys.prefix, sys.exec_prefix])
20198 f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	146 # We need to supplement the list of prefixes for the search to work
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	147 # when run from within a virtualenv.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	148 for mod in (BaseHTTPServer, zlib):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	149 try:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	150 # Not all module objects have a __file__ attribute.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	151 filename = mod.__file__
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	152 except AttributeError:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	153 continue
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	154 dirname = os.path.dirname(filename)
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	155 for prefix in stdlib_prefixes:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	156 if dirname.startswith(prefix):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	157 # Then this directory is redundant.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	158 break
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	159 else:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	160 stdlib_prefixes.add(dirname)
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	161 for libpath in sys.path:
20201 bc3b48b0f5c8 import-checker: suppress check-code about any() Augie Fackler <raf@durin42.com> parents: 20200 diff changeset	162 # We want to walk everything in sys.path that starts with
bc3b48b0f5c8 import-checker: suppress check-code about any() Augie Fackler <raf@durin42.com> parents: 20200 diff changeset	163 # something in stdlib_prefixes. check-code suppressed because
bc3b48b0f5c8 import-checker: suppress check-code about any() Augie Fackler <raf@durin42.com> parents: 20200 diff changeset	164 # the ast module used by this script implies the availability
bc3b48b0f5c8 import-checker: suppress check-code about any() Augie Fackler <raf@durin42.com> parents: 20200 diff changeset	165 # of any().
20238 81e905790b30 check-code: do not skip entire file, skip only one match instead Simon Heimberg <simohe@besonet.ch> parents: 20201 diff changeset	166 if not any(libpath.startswith(p) for p in stdlib_prefixes): # no-py24
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	167 continue
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	168 if 'site-packages' in libpath:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	169 continue
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	170 for top, dirs, files in os.walk(libpath):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	171 for name in files:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	172 if name == '__init__.py':
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	173 continue
24668 81873bb2a01d import-checker: allow .pyd based stdlib modules Matt Harbison <matt_harbison@yahoo.com>* parents: 24491 diff changeset	174 if not (name.endswith('.py') or name.endswith('.so')
81873bb2a01d import-checker: allow .pyd based stdlib modules Matt Harbison <matt_harbison@yahoo.com>* parents: 24491 diff changeset	175 or name.endswith('.pyd')):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	176 continue
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	177 full_path = os.path.join(top, name)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	178 if 'site-packages' in full_path:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	179 continue
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	180 rel_path = full_path[len(libpath) + 1:]
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	181 mod = dotted_name_of_path(rel_path)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	182 yield mod
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	183
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	184 stdlib_modules = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	185
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	186 def imported_modules(source, modulename, localmods, ignore_nested=False):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	187 """Given the source of a file as a string, yield the names
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	188 imported by that file.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	189
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	190 Args:
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	191 source: The python source to examine as a string.
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	192 modulename: of specified python source (may have `__init__`)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	193 localmods: dict of locally defined module names (may have `__init__`)
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	194 ignore_nested: If true, import statements that do not start in
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	195 column zero will be ignored.
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	196
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	197 Returns:
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	198 A list of absolute module names imported by the given source.
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	199
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	200 >>> modulename = 'foo.xxx'
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	201 >>> localmods = {'foo.__init__': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	202 ... 'foo.foo1': True, 'foo.foo2': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	203 ... 'foo.bar.__init__': True, 'foo.bar.bar1': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	204 ... 'baz.__init__': True, 'baz.baz1': True }
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	205 >>> # standard library (= not locally defined ones)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	206 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	207 ... 'from stdlib1 import foo, bar; import stdlib2',
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	208 ... modulename, localmods))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	209 []
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	210 >>> # relative importing
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	211 >>> sorted(imported_modules(
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	212 ... 'import foo1; from bar import bar1',
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	213 ... modulename, localmods))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	214 ['foo.bar.__init__', 'foo.bar.bar1', 'foo.foo1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	215 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	216 ... 'from bar.bar1 import name1, name2, name3',
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	217 ... modulename, localmods))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	218 ['foo.bar.bar1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	219 >>> # absolute importing
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	220 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	221 ... 'from baz import baz1, name1',
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	222 ... modulename, localmods))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	223 ['baz.__init__', 'baz.baz1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	224 >>> # mixed importing, even though it shouldn't be recommended
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	225 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	226 ... 'import stdlib, foo1, baz',
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	227 ... modulename, localmods))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	228 ['baz.__init__', 'foo.foo1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	229 >>> # ignore_nested
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	230 >>> sorted(imported_modules(
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	231 ... '''import foo
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	232 ... def wat():
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	233 ... import bar
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	234 ... ''', modulename, localmods))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	235 ['foo.__init__', 'foo.bar.__init__']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	236 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	237 ... '''import foo
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	238 ... def wat():
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	239 ... import bar
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	240 ... ''', modulename, localmods, ignore_nested=True))
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	241 ['foo.__init__']
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	242 """
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	243 fromlocal = fromlocalfunc(modulename, localmods)
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	244 for node in ast.walk(ast.parse(source)):
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	245 if ignore_nested and getattr(node, 'col_offset', 0) > 0:
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	246 continue
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	247 if isinstance(node, ast.Import):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	248 for n in node.names:
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	249 found = fromlocal(n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	250 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	251 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	252 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	253 yield found[1]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	254 elif isinstance(node, ast.ImportFrom):
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	255 found = fromlocal(node.module, node.level)
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	256 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	257 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	258 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	259
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	260 absname, dottedpath, hassubmod = found
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	261 yield dottedpath
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	262 if not hassubmod:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	263 # examination of "node.names" should be redundant
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	264 # e.g.: from mercurial.node import nullid, nullrev
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	265 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	266
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	267 prefix = absname + '.'
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	268 for n in node.names:
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	269 found = fromlocal(prefix + n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	270 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	271 # this should be a function or a property of "node.module"
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	272 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	273 yield found[1]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	274
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	275 def verify_stdlib_on_own_line(source):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	276 """Given some python source, verify that stdlib imports are done
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	277 in separate statements from relative local module imports.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	278
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	279 Observing this limitation is important as it works around an
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	280 annoying lib2to3 bug in relative import rewrites:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	281 http://bugs.python.org/issue19510.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	282
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	283 >>> list(verify_stdlib_on_own_line('import sys, foo'))
20386 a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	284 ['mixed imports\\n stdlib: sys\\n relative: foo']
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	285 >>> list(verify_stdlib_on_own_line('import sys, os'))
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	286 []
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	287 >>> list(verify_stdlib_on_own_line('import foo, bar'))
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	288 []
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	289 """
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	290 for node in ast.walk(ast.parse(source)):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	291 if isinstance(node, ast.Import):
20386 a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	292 from_stdlib = {False: [], True: []}
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	293 for n in node.names:
20386 a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	294 from_stdlib[n.name in stdlib_modules].append(n.name)
a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	295 if from_stdlib[True] and from_stdlib[False]:
a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	296 yield ('mixed imports\n stdlib: %s\n relative: %s' %
a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	297 (', '.join(sorted(from_stdlib[True])),
a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	298 ', '.join(sorted(from_stdlib[False]))))
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	299
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	300 class CircularImport(Exception):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	301 pass
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	302
24490 fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	303 def checkmod(mod, imports):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	304 shortest = {}
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	305 visit = [[mod]]
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	306 while visit:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	307 path = visit.pop(0)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	308 for i in sorted(imports.get(path[-1], [])):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	309 if len(path) < shortest.get(i, 1000):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	310 shortest[i] = len(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	311 if i in path:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	312 if i == path[0]:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	313 raise CircularImport(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	314 continue
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	315 visit.append(path + [i])
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	316
20038 c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	317 def rotatecycle(cycle):
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	318 """arrange a cycle so that the lexicographically first module listed first
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	319
24488 4b3fc46097f7 import-checker: drop duplicate element from cycle Matt Mackall <mpm@selenic.com> parents: 24487 diff changeset	320 >>> rotatecycle(['foo', 'bar'])
20038 c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	321 ['bar', 'foo', 'bar']
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	322 """
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	323 lowest = min(cycle)
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	324 idx = cycle.index(lowest)
24488 4b3fc46097f7 import-checker: drop duplicate element from cycle Matt Mackall <mpm@selenic.com> parents: 24487 diff changeset	325 return cycle[idx:] + cycle[:idx] + [lowest]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	326
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	327 def find_cycles(imports):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	328 """Find cycles in an already-loaded import graph.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	329
25175 10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	330 All module names recorded in `imports` should be absolute one.
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	331
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	332 >>> imports = {'top.foo': ['top.bar', 'os.path', 'top.qux'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	333 ... 'top.bar': ['top.baz', 'sys'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	334 ... 'top.baz': ['top.foo'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	335 ... 'top.qux': ['top.foo']}
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	336 >>> print '\\n'.join(sorted(find_cycles(imports)))
24487 642d245ff537 import-checker: fix rotatecycle Matt Mackall <mpm@selenic.com> parents: 22975 diff changeset	337 top.bar -> top.baz -> top.foo -> top.bar
642d245ff537 import-checker: fix rotatecycle Matt Mackall <mpm@selenic.com> parents: 22975 diff changeset	338 top.foo -> top.qux -> top.foo
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	339 """
24491 784b278b349c import-checker: rotatecycle is actually the canonical cycle key Matt Mackall <mpm@selenic.com> parents: 24490 diff changeset	340 cycles = set()
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	341 for mod in sorted(imports.iterkeys()):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	342 try:
24490 fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	343 checkmod(mod, imports)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25175 diff changeset	344 except CircularImport as e:
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	345 cycle = e.args[0]
24491 784b278b349c import-checker: rotatecycle is actually the canonical cycle key Matt Mackall <mpm@selenic.com> parents: 24490 diff changeset	346 cycles.add(" -> ".join(rotatecycle(cycle)))
784b278b349c import-checker: rotatecycle is actually the canonical cycle key Matt Mackall <mpm@selenic.com> parents: 24490 diff changeset	347 return cycles
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	348
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	349 def _cycle_sortkey(c):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	350 return len(c), c
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	351
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	352 def main(argv):
25063 723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	353 if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	354 print 'Usage: %s {-\|file [file] [file] ...}'
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	355 return 1
25063 723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	356 if argv[1] == '-':
723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	357 argv = argv[:1]
723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	358 argv.extend(l.rstrip() for l in sys.stdin.readlines())
25064 3bbbadf69d0a import-checker: loop to get list of locally defined modules at first FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25063 diff changeset	359 localmods = {}
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	360 used_imports = {}
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	361 any_errors = False
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	362 for source_path in argv[1:]:
25064 3bbbadf69d0a import-checker: loop to get list of locally defined modules at first FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25063 diff changeset	363 modname = dotted_name_of_path(source_path, trimpure=True)
3bbbadf69d0a import-checker: loop to get list of locally defined modules at first FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25063 diff changeset	364 localmods[modname] = source_path
3bbbadf69d0a import-checker: loop to get list of locally defined modules at first FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25063 diff changeset	365 for modname, source_path in sorted(localmods.iteritems()):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	366 f = open(source_path)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	367 src = f.read()
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	368 used_imports[modname] = sorted(
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	369 imported_modules(src, modname, localmods, ignore_nested=True))
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	370 for error in verify_stdlib_on_own_line(src):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	371 any_errors = True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	372 print source_path, error
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	373 f.close()
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	374 cycles = find_cycles(used_imports)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	375 if cycles:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	376 firstmods = set()
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	377 for c in sorted(cycles, key=_cycle_sortkey):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	378 first = c.split()[0]
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	379 # As a rough cut, ignore any cycle that starts with the
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	380 # same module as some other cycle. Otherwise we see lots
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	381 # of cycles that are effectively duplicates.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	382 if first in firstmods:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	383 continue
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	384 print 'Import cycle:', c
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	385 firstmods.add(first)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	386 any_errors = True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	387 return not any_errors
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	388
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	389 if __name__ == '__main__':
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	390 sys.exit(int(main(sys.argv)))

Mercurial > public > mercurial-scm > hg

annotate contrib/import-checker.py @ 25701:1f88c0f6ff5a