mercurial-scm/hg: contrib/import-checker.py annotate

annotate contrib/import-checker.py @ 43084:c2e284cee333

import-checker: allow symbol imports from mercurial.pycompat Currently, the source transformer inserts `from mercurial.pycompat import delattr, getattr, hasattr, setattr, open, unicode` to the top of every file. As part of getting rid of the source transformer, we'll need to have source code call these wrappers directly. Rather than rewrite all call sites to call pycompat.*, I think it makes sense to import needed symbols via explicit imports. That requires loosening the import checker to allow this. Differential Revision: https://phab.mercurial-scm.org/D7004

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Sun, 06 Oct 2019 13:17:19 -0400
parents	57875cf423c9
children	a8454e846736

rev	line source
26954 f804bf27439b import-checker: make it executable for convenience Yuya Nishihara <yuya@tcha.org> parents: 26781 diff changeset	1 #!/usr/bin/env python
f804bf27439b import-checker: make it executable for convenience Yuya Nishihara <yuya@tcha.org> parents: 26781 diff changeset	2
28703 a274c4f9087a py3: use print_function in import-checker timeless <timeless@mozdev.org> parents: 28702 diff changeset	3 from __future__ import absolute_import, print_function
28702 e44f671018e3 py3: use absolute_import in import-checker timeless <timeless@mozdev.org> parents: 28700 diff changeset	4
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	5 import ast
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	6 import collections
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	7 import os
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	8 import sys
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	9
20198 f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	10 # Import a minimal set of stdlib modules needed for list_stdlib_modules()
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	11 # to work when run from a virtualenv. The modules were chosen empirically
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	12 # so that the return value matches the return value without virtualenv.
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	13 if True: # disable lexical sorting checks
33876 998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	14 try:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	15 import BaseHTTPServer as basehttpserver
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	16 except ImportError:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	17 basehttpserver = None
29211 b42c2a66a698 py3: make contrib/import-checker.py get along with itself Yuya Nishihara <yuya@tcha.org> parents: 29208 diff changeset	18 import zlib
20198 f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	19
40095 7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	20 import testparseutil
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	21
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	22 # Whitelist of modules that symbols can be directly imported from.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	23 allowsymbolimports = (
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	24 '__future__',
33919 5ed0be4d9df9 contrib: add bzrlib to list of packages from which we import symbols Augie Fackler <raf@durin42.com> parents: 33915 diff changeset	25 'bzrlib',
33915 2d64b2f1787b contrib: allow symbol imports from hgclient for tests Augie Fackler <raf@durin42.com> parents: 33897 diff changeset	26 'hgclient',
33893 c9cf69d0c3b9 contrib: allow importing "symbols" from mercurial Augie Fackler <raf@durin42.com> parents: 33890 diff changeset	27 'mercurial',
27018 e5be48dd8215 import-checker: allow symbol imports from hgweb.common and .request Yuya Nishihara <yuya@tcha.org> parents: 26965 diff changeset	28 'mercurial.hgweb.common',
e5be48dd8215 import-checker: allow symbol imports from hgweb.common and .request Yuya Nishihara <yuya@tcha.org> parents: 26965 diff changeset	29 'mercurial.hgweb.request',
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	30 'mercurial.i18n',
42813 268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 42728 diff changeset	31 'mercurial.interfaces',
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	32 'mercurial.node',
43084 c2e284cee333 import-checker: allow symbol imports from mercurial.pycompat Gregory Szorc <gregory.szorc@gmail.com> parents: 43075 diff changeset	33 'mercurial.pycompat',
39329 729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 38797 diff changeset	34 # for revlog to re-export constant to extensions
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 38797 diff changeset	35 'mercurial.revlogutils.constants',
42728 ca5ca3badd3c flagutil: create a `mercurial.revlogutils.flagutil` module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42388 diff changeset	36 'mercurial.revlogutils.flagutil',
32507 95085d747db8 import-checker: allow importing symbols from pure modules Yuya Nishihara <yuya@tcha.org> parents: 32420 diff changeset	37 # for cffi modules to re-export pure functions
95085d747db8 import-checker: allow importing symbols from pure modules Yuya Nishihara <yuya@tcha.org> parents: 32420 diff changeset	38 'mercurial.pure.base85',
95085d747db8 import-checker: allow importing symbols from pure modules Yuya Nishihara <yuya@tcha.org> parents: 32420 diff changeset	39 'mercurial.pure.bdiff',
95085d747db8 import-checker: allow importing symbols from pure modules Yuya Nishihara <yuya@tcha.org> parents: 32420 diff changeset	40 'mercurial.pure.mpatch',
95085d747db8 import-checker: allow importing symbols from pure modules Yuya Nishihara <yuya@tcha.org> parents: 32420 diff changeset	41 'mercurial.pure.osutil',
95085d747db8 import-checker: allow importing symbols from pure modules Yuya Nishihara <yuya@tcha.org> parents: 32420 diff changeset	42 'mercurial.pure.parsers',
34395 41401f502c83 tests: disable lints on mercurial/thirdparty Siddharth Agarwal <sid0@fb.com> parents: 34038 diff changeset	43 # third-party imports should be directly imported
41401f502c83 tests: disable lints on mercurial/thirdparty Siddharth Agarwal <sid0@fb.com> parents: 34038 diff changeset	44 'mercurial.thirdparty',
38797 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 37711 diff changeset	45 'mercurial.thirdparty.attr',
37180 922b3fae9c7d setup: register zope.interface packages and compile C extension Gregory Szorc <gregory.szorc@gmail.com> parents: 34395 diff changeset	46 'mercurial.thirdparty.zope',
922b3fae9c7d setup: register zope.interface packages and compile C extension Gregory Szorc <gregory.szorc@gmail.com> parents: 34395 diff changeset	47 'mercurial.thirdparty.zope.interface',
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	48 )
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	49
32419 d02888308235 import-checker: add a way to directly import certain symbols Siddharth Agarwal <sid0@fb.com> parents: 32374 diff changeset	50 # Whitelist of symbols that can be directly imported.
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	51 directsymbols = ('demandimport',)
32419 d02888308235 import-checker: add a way to directly import certain symbols Siddharth Agarwal <sid0@fb.com> parents: 32374 diff changeset	52
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	53 # Modules that must be aliased because they are commonly confused with
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	54 # common variables and can create aliasing and readability issues.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	55 requirealias = {
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	56 'ui': 'uimod',
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	57 }
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	58
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	59
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	60 def usingabsolute(root):
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	61 """Whether absolute imports are being used."""
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	62 if sys.version_info[0] >= 3:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	63 return True
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	64
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	65 for node in ast.walk(root):
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	66 if isinstance(node, ast.ImportFrom):
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	67 if node.module == '__future__':
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	68 for n in node.names:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	69 if n.name == 'absolute_import':
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	70 return True
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	71
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	72 return False
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	73
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	74
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	75 def walklocal(root):
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	76 """Recursively yield all descendant nodes but not in a different scope"""
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	77 todo = collections.deque(ast.iter_child_nodes(root))
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	78 yield root, False
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	79 while todo:
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	80 node = todo.popleft()
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	81 newscope = isinstance(node, ast.FunctionDef)
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	82 if not newscope:
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	83 todo.extend(ast.iter_child_nodes(node))
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	84 yield node, newscope
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	85
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	86
32374 194b0f781132 import-checker: drop workaround for pure modules Yuya Nishihara <yuya@tcha.org> parents: 32372 diff changeset	87 def dotted_name_of_path(path):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	88 """Given a relative path to a source file, return its dotted module name.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	89
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	90 >>> dotted_name_of_path('mercurial/error.py')
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	91 'mercurial.error'
20383 4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	92 >>> dotted_name_of_path('zlibmodule.so')
4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	93 'zlib'
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	94 """
27620 0c60843b55b5 import-checker: normalize directory separator to get module name on Windows Yuya Nishihara <yuya@tcha.org> parents: 27520 diff changeset	95 parts = path.replace(os.sep, '/').split('/')
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	96 parts[-1] = parts[-1].split('.', 1)[0] # remove .py and .so and .ARCH.so
20383 4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	97 if parts[-1].endswith('module'):
4990abb4729d import-checker: fix names of dynamically loaded modules Mads Kiilerich <madski@unity3d.com> parents: 20238 diff changeset	98 parts[-1] = parts[-1][:-6]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	99 return '.'.join(parts)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	100
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	101
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	102 def fromlocalfunc(modulename, localmods):
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	103 """Get a function to examine which locally defined module the
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	104 target source imports via a specified name.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	105
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	106 `modulename` is an `dotted_name_of_path()`-ed source file path,
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	107 which may have `.__init__` at the end of it, of the target source.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	108
32508 4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	109 `localmods` is a set of absolute `dotted_name_of_path()`-ed source file
4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	110 paths of locally defined (= Mercurial specific) modules.
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	111
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	112 This function assumes that module names not existing in
26781 1aee2ab0f902 spelling: trivial spell checking Mads Kiilerich <madski@unity3d.com> parents: 26221 diff changeset	113 `localmods` are from the Python standard library.
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	114
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	115 This function returns the function, which takes `name` argument,
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	116 and returns `(absname, dottedpath, hassubmod)` tuple if `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	117 matches against locally defined module. Otherwise, it returns
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	118 False.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	119
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	120 It is assumed that `name` doesn't have `.__init__`.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	121
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	122 `absname` is an absolute module name of specified `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	123 (e.g. "hgext.convert"). This can be used to compose prefix for sub
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	124 modules or so.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	125
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	126 `dottedpath` is a `dotted_name_of_path()`-ed source file path
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	127 (e.g. "hgext.convert.__init__") of `name`. This is used to look
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	128 module up in `localmods` again.
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	129
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	130 `hassubmod` is whether it may have sub modules under it (for
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	131 convenient, even though this is also equivalent to "absname !=
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	132 dottednpath")
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	133
32508 4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	134 >>> localmods = {'foo.__init__', 'foo.foo1',
4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	135 ... 'foo.bar.__init__', 'foo.bar.bar1',
4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	136 ... 'baz.__init__', 'baz.baz1'}
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	137 >>> fromlocal = fromlocalfunc('foo.xxx', localmods)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	138 >>> # relative
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	139 >>> fromlocal('foo1')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	140 ('foo.foo1', 'foo.foo1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	141 >>> fromlocal('bar')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	142 ('foo.bar', 'foo.bar.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	143 >>> fromlocal('bar.bar1')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	144 ('foo.bar.bar1', 'foo.bar.bar1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	145 >>> # absolute
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	146 >>> fromlocal('baz')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	147 ('baz', 'baz.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	148 >>> fromlocal('baz.baz1')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	149 ('baz.baz1', 'baz.baz1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	150 >>> # unknown = maybe standard library
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	151 >>> fromlocal('os')
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	152 False
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	153 >>> fromlocal(None, 1)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	154 ('foo', 'foo.__init__', True)
29122 660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	155 >>> fromlocal('foo1', 1)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	156 ('foo.foo1', 'foo.foo1', False)
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	157 >>> fromlocal2 = fromlocalfunc('foo.xxx.yyy', localmods)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	158 >>> fromlocal2(None, 2)
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	159 ('foo', 'foo.__init__', True)
29122 660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	160 >>> fromlocal2('bar2', 1)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	161 False
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	162 >>> fromlocal2('bar', 2)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	163 ('foo.bar', 'foo.bar.__init__', True)
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	164 """
33890 3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	165 if not isinstance(modulename, str):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	166 modulename = modulename.decode('ascii')
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	167 prefix = '.'.join(modulename.split('.')[:-1])
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	168 if prefix:
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	169 prefix += '.'
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	170
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	171 def fromlocal(name, level=0):
29374 7712fcde2d56 import-checker: increase portability for python 2.6.x FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29234 diff changeset	172 # name is false value when relative imports are used.
7712fcde2d56 import-checker: increase portability for python 2.6.x FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29234 diff changeset	173 if not name:
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	174 # If relative imports are used, level must not be absolute.
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	175 assert level > 0
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	176 candidates = ['.'.join(modulename.split('.')[:-level])]
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	177 else:
29122 660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	178 if not level:
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	179 # Check relative name first.
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	180 candidates = [prefix + name, name]
660d8d4ec7aa import-checker: recognize relative imports from parents of current package liscju <piotr.listkiewicz@gmail.com> parents: 28922 diff changeset	181 else:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	182 candidates = [
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	183 '.'.join(modulename.split('.')[:-level]) + '.' + name
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	184 ]
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	185
1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	186 for n in candidates:
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	187 if n in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	188 return (n, n, False)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	189 dottedpath = n + '.__init__'
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	190 if dottedpath in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	191 return (n, dottedpath, True)
7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	192 return False
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	193
25173 7358b5d9991e import-checker: add utility to examine what module is imported easily FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25064 diff changeset	194 return fromlocal
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	195
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	196
32509 a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	197 def populateextmods(localmods):
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	198 """Populate C extension modules based on pure modules"""
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	199 newlocalmods = set(localmods)
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	200 for n in localmods:
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	201 if n.startswith('mercurial.pure.'):
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	202 m = n[len('mercurial.pure.') :]
32509 a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	203 newlocalmods.add('mercurial.cext.' + m)
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	204 newlocalmods.add('mercurial.cffi._' + m)
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	205 return newlocalmods
a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	206
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	207
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	208 def list_stdlib_modules():
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	209 """List the modules present in the stdlib.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	210
33876 998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	211 >>> py3 = sys.version_info[0] >= 3
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	212 >>> mods = set(list_stdlib_modules())
33876 998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	213 >>> 'BaseHTTPServer' in mods or py3
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	214 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	215
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	216 os.path isn't really a module, so it's missing:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	217
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	218 >>> 'os.path' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	219 False
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	220
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	221 sys requires special treatment, because it's baked into the
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	222 interpreter, but it should still appear:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	223
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	224 >>> 'sys' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	225 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	226
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	227 >>> 'collections' in mods
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	228 True
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	229
33876 998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	230 >>> 'cStringIO' in mods or py3
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	231 True
29395 4c8026babe8c import-checker: ensure cffi is always a system module Augie Fackler <raf@durin42.com> parents: 29374 diff changeset	232
4c8026babe8c import-checker: ensure cffi is always a system module Augie Fackler <raf@durin42.com> parents: 29374 diff changeset	233 >>> 'cffi' in mods
4c8026babe8c import-checker: ensure cffi is always a system module Augie Fackler <raf@durin42.com> parents: 29374 diff changeset	234 True
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	235 """
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	236 for m in sys.builtin_module_names:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	237 yield m
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	238 # These modules only exist on windows, but we should always
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	239 # consider them stdlib.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	240 for m in ['msvcrt', '_winreg']:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	241 yield m
33894 c856cb1c29be contrib: inform import checker that __builtin__ is a thing Augie Fackler <raf@durin42.com> parents: 33893 diff changeset	242 yield '__builtin__'
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	243 yield 'builtins' # python3 only
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	244 yield 'importlib.abc' # python3 only
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	245 yield 'importlib.machinery' # python3 only
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	246 yield 'importlib.util' # python3 only
24669 fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules Matt Harbison <matt_harbison@yahoo.com> parents: 24668 diff changeset	247 for m in 'fcntl', 'grp', 'pwd', 'termios': # Unix only
fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules Matt Harbison <matt_harbison@yahoo.com> parents: 24668 diff changeset	248 yield m
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	249 for m in 'cPickle', 'datetime': # in Python (not C) on PyPy
28713 806d260c6f3b tests: fix builtin module test on pypy Maciej Fijalkowski <fijall@gmail.com> parents: 28704 diff changeset	250 yield m
29395 4c8026babe8c import-checker: ensure cffi is always a system module Augie Fackler <raf@durin42.com> parents: 29374 diff changeset	251 for m in ['cffi']:
4c8026babe8c import-checker: ensure cffi is always a system module Augie Fackler <raf@durin42.com> parents: 29374 diff changeset	252 yield m
32291 bd872f64a8ba cleanup: use set literals Martin von Zweigbergk <martinvonz@google.com> parents: 32212 diff changeset	253 stdlib_prefixes = {sys.prefix, sys.exec_prefix}
20198 f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	254 # We need to supplement the list of prefixes for the search to work
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	255 # when run from within a virtualenv.
33876 998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	256 for mod in (basehttpserver, zlib):
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	257 if mod is None:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker Augie Fackler <raf@durin42.com> parents: 32595 diff changeset	258 continue
20198 f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	259 try:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	260 # Not all module objects have a __file__ attribute.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	261 filename = mod.__file__
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	262 except AttributeError:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	263 continue
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	264 dirname = os.path.dirname(filename)
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	265 for prefix in stdlib_prefixes:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	266 if dirname.startswith(prefix):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	267 # Then this directory is redundant.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	268 break
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	269 else:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129) Chris Jerdonek <chris.jerdonek@gmail.com> parents: 20197 diff changeset	270 stdlib_prefixes.add(dirname)
40687 dd028bca9221 tests: make test-check-module-imports more robust Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 40481 diff changeset	271 sourceroot = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	272 for libpath in sys.path:
40687 dd028bca9221 tests: make test-check-module-imports more robust Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 40481 diff changeset	273 # We want to walk everything in sys.path that starts with something in
dd028bca9221 tests: make test-check-module-imports more robust Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 40481 diff changeset	274 # stdlib_prefixes, but not directories from the hg sources.
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	275 if os.path.abspath(libpath).startswith(sourceroot) or not any(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	276 libpath.startswith(p) for p in stdlib_prefixes
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	277 ):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	278 continue
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	279 for top, dirs, files in os.walk(libpath):
25733 f99c066f5f9a import-checker: recurse into subtree of sys.path only if __init__.py exists Yuya Nishihara <yuya@tcha.org> parents: 25731 diff changeset	280 for i, d in reversed(list(enumerate(dirs))):
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	281 if (
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	282 not os.path.exists(os.path.join(top, d, '__init__.py'))
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	283 or top == libpath
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	284 and d in ('hgdemandimport', 'hgext', 'mercurial')
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	285 ):
25733 f99c066f5f9a import-checker: recurse into subtree of sys.path only if __init__.py exists Yuya Nishihara <yuya@tcha.org> parents: 25731 diff changeset	286 del dirs[i]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	287 for name in files:
26221 ae65b1b4cb46 import-checker: use modern .endswith for multiple suffixes Augie Fackler <augie@google.com> parents: 26166 diff changeset	288 if not name.endswith(('.py', '.so', '.pyc', '.pyo', '.pyd')):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	289 continue
27621 39845b064041 import-checker: list package directory as stdlib module Yuya Nishihara <yuya@tcha.org> parents: 27620 diff changeset	290 if name.startswith('__init__.py'):
39845b064041 import-checker: list package directory as stdlib module Yuya Nishihara <yuya@tcha.org> parents: 27620 diff changeset	291 full_path = top
39845b064041 import-checker: list package directory as stdlib module Yuya Nishihara <yuya@tcha.org> parents: 27620 diff changeset	292 else:
39845b064041 import-checker: list package directory as stdlib module Yuya Nishihara <yuya@tcha.org> parents: 27620 diff changeset	293 full_path = os.path.join(top, name)
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	294 rel_path = full_path[len(libpath) + 1 :]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	295 mod = dotted_name_of_path(rel_path)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	296 yield mod
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	297
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	298
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	299 stdlib_modules = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	300
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	301
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	302 def imported_modules(source, modulename, f, localmods, ignore_nested=False):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	303 """Given the source of a file as a string, yield the names
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	304 imported by that file.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	305
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	306 Args:
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	307 source: The python source to examine as a string.
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	308 modulename: of specified python source (may have `__init__`)
32508 4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	309 localmods: set of locally defined module names (may have `__init__`)
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	310 ignore_nested: If true, import statements that do not start in
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	311 column zero will be ignored.
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	312
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	313 Returns:
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	314 A list of absolute module names imported by the given source.
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	315
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	316 >>> f = 'foo/xxx.py'
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	317 >>> modulename = 'foo.xxx'
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	318 >>> localmods = {'foo.__init__': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	319 ... 'foo.foo1': True, 'foo.foo2': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	320 ... 'foo.bar.__init__': True, 'foo.bar.bar1': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	321 ... 'baz.__init__': True, 'baz.baz1': True }
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	322 >>> # standard library (= not locally defined ones)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	323 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	324 ... 'from stdlib1 import foo, bar; import stdlib2',
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	325 ... modulename, f, localmods))
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	326 []
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	327 >>> # relative importing
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	328 >>> sorted(imported_modules(
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	329 ... 'import foo1; from bar import bar1',
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	330 ... modulename, f, localmods))
26964 5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	331 ['foo.bar.bar1', 'foo.foo1']
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	332 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	333 ... 'from bar.bar1 import name1, name2, name3',
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	334 ... modulename, f, localmods))
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	335 ['foo.bar.bar1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	336 >>> # absolute importing
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	337 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	338 ... 'from baz import baz1, name1',
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	339 ... modulename, f, localmods))
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	340 ['baz.__init__', 'baz.baz1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	341 >>> # mixed importing, even though it shouldn't be recommended
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	342 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	343 ... 'import stdlib, foo1, baz',
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	344 ... modulename, f, localmods))
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	345 ['baz.__init__', 'foo.foo1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	346 >>> # ignore_nested
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	347 >>> sorted(imported_modules(
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	348 ... '''import foo
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	349 ... def wat():
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	350 ... import bar
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	351 ... ''', modulename, f, localmods))
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	352 ['foo.__init__', 'foo.bar.__init__']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	353 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	354 ... '''import foo
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	355 ... def wat():
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	356 ... import bar
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	357 ... ''', modulename, f, localmods, ignore_nested=True))
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	358 ['foo.__init__']
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	359 """
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	360 fromlocal = fromlocalfunc(modulename, localmods)
28921 02ee31a50002 import-checker: track filenames for SyntaxErrors timeless <timeless@mozdev.org> parents: 28920 diff changeset	361 for node in ast.walk(ast.parse(source, f)):
20037 957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	362 if ignore_nested and getattr(node, 'col_offset', 0) > 0:
957b43371928 import-checker: ignore nested imports Augie Fackler <raf@durin42.com> parents: 20036 diff changeset	363 continue
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	364 if isinstance(node, ast.Import):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	365 for n in node.names:
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	366 found = fromlocal(n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	367 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	368 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	369 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	370 yield found[1]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	371 elif isinstance(node, ast.ImportFrom):
25701 1f88c0f6ff5a import-checker: resolve relative imports Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	372 found = fromlocal(node.module, node.level)
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	373 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	374 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	375 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	376
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	377 absname, dottedpath, hassubmod = found
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	378 if not hassubmod:
26964 5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	379 # "dottedpath" is not a package; must be imported
5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	380 yield dottedpath
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	381 # examination of "node.names" should be redundant
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	382 # e.g.: from mercurial.node import nullid, nullrev
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	383 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	384
26964 5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	385 modnotfound = False
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	386 prefix = absname + '.'
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	387 for n in node.names:
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	388 found = fromlocal(prefix + n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	389 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	390 # this should be a function or a property of "node.module"
26964 5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	391 modnotfound = True
25174 86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	392 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25173 diff changeset	393 yield found[1]
26964 5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	394 if modnotfound:
5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	395 # "dottedpath" is a package, but imported because of non-module
5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	396 # lookup
5abba2c92da3 import-checker: allow import of child modules from package root Yuya Nishihara <yuya@tcha.org> parents: 26956 diff changeset	397 yield dottedpath
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	398
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	399
27272 69308357ecd1 import-checker: allow absolute imports of sub modules from local packages Yuya Nishihara <yuya@tcha.org> parents: 27018 diff changeset	400 def verify_import_convention(module, source, localmods):
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	401 """Verify imports match our established coding convention.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	402
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	403 We have 2 conventions: legacy and modern. The modern convention is in
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	404 effect when using absolute imports.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	405
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	406 The legacy convention only looks for mixed imports. The modern convention
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	407 is much more thorough.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	408 """
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	409 root = ast.parse(source)
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	410 absolute = usingabsolute(root)
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	411
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	412 if absolute:
27272 69308357ecd1 import-checker: allow absolute imports of sub modules from local packages Yuya Nishihara <yuya@tcha.org> parents: 27018 diff changeset	413 return verify_modern_convention(module, root, localmods)
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	414 else:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	415 return verify_stdlib_on_own_line(root)
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	416
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	417
27272 69308357ecd1 import-checker: allow absolute imports of sub modules from local packages Yuya Nishihara <yuya@tcha.org> parents: 27018 diff changeset	418 def verify_modern_convention(module, root, localmods, root_col_offset=0):
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	419 """Verify a file conforms to the modern import convention rules.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	420
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	421 The rules of the modern convention are:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	422
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	423 * Ordering is stdlib followed by local imports. Each group is lexically
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	424 sorted.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	425 * Importing multiple modules via "import X, Y" is not allowed: use
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	426 separate import statements.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	427 * Importing multiple modules via "from X import ..." is allowed if using
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	428 parenthesis and one entry per line.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	429 * Only 1 relative import statement per import level ("from .", "from ..")
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	430 is allowed.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	431 * Relative imports from higher levels must occur before lower levels. e.g.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	432 "from .." must be before "from .".
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	433 * Imports from peer packages should use relative import (e.g. do not
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	434 "import mercurial.foo" from a "mercurial.*" module).
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	435 * Symbols can only be imported from specific modules (see
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	436 `allowsymbolimports`). For other modules, first import the module then
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	437 assign the symbol to a module-level variable. In addition, these imports
29208 cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules Yuya Nishihara <yuya@tcha.org> parents: 29207 diff changeset	438 must be performed before other local imports. This rule only
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	439 applies to import statements outside of any blocks.
34038 bc2535238de2 import-checker: allow relative import a module being checked Jun Wu <quark@fb.com> parents: 33920 diff changeset	440 * Relative imports from the standard library are not allowed, unless that
bc2535238de2 import-checker: allow relative import a module being checked Jun Wu <quark@fb.com> parents: 33920 diff changeset	441 library is also a local module.
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	442 * Certain modules must be aliased to alternate names to avoid aliasing
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	443 and readability problems. See `requirealias`.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	444 """
33890 3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	445 if not isinstance(module, str):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	446 module = module.decode('ascii')
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	447 topmodule = module.split('.')[0]
27272 69308357ecd1 import-checker: allow absolute imports of sub modules from local packages Yuya Nishihara <yuya@tcha.org> parents: 27018 diff changeset	448 fromlocal = fromlocalfunc(module, localmods)
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	449
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	450 # Whether a local/non-stdlib import has been performed.
28330 f3fb24e36d61 import-checker: report local with stdlib late warning timeless <timeless@mozdev.org> parents: 27621 diff changeset	451 seenlocal = None
29208 cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules Yuya Nishihara <yuya@tcha.org> parents: 29207 diff changeset	452 # Whether a local/non-stdlib, non-symbol import has been seen.
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules Yuya Nishihara <yuya@tcha.org> parents: 29207 diff changeset	453 seennonsymbollocal = False
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	454 # The last name to be imported (for sorting).
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	455 lastname = None
30590 74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 29395 diff changeset	456 laststdlib = None
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	457 # Relative import levels encountered so far.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	458 seenlevels = set()
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	459
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	460 for node, newscope in walklocal(root):
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	461
26955 c4114e335b49 import-checker: extract function to generate a formatted warning Yuya Nishihara <yuya@tcha.org> parents: 26954 diff changeset	462 def msg(fmt, *args):
26956 4b56214ebb7a import-checker: include lineno in warning message Yuya Nishihara <yuya@tcha.org> parents: 26955 diff changeset	463 return (fmt % args, node.lineno)
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	464
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	465 if newscope:
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	466 # Check for local imports in function
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	467 for r in verify_modern_convention(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	468 module, node, localmods, node.col_offset + 4
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	469 ):
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	470 yield r
1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	471 elif isinstance(node, ast.Import):
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	472 # Disallow "import foo, bar" and require separate imports
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	473 # for each module.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	474 if len(node.names) > 1:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	475 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	476 'multiple imported names: %s',
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	477 ', '.join(n.name for n in node.names),
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	478 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	479
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	480 name = node.names[0].name
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	481 asname = node.names[0].asname
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	482
30590 74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 29395 diff changeset	483 stdlib = name in stdlib_modules
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 29395 diff changeset	484
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	485 # Ignore sorting rules on imports inside blocks.
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	486 if node.col_offset == root_col_offset:
30590 74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 29395 diff changeset	487 if lastname and name < lastname and laststdlib == stdlib:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	488 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	489 'imports not lexically sorted: %s < %s', name, lastname
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	490 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	491
30590 74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 29395 diff changeset	492 lastname = name
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 29395 diff changeset	493 laststdlib = stdlib
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	494
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	495 # stdlib imports should be before local imports.
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	496 if stdlib and seenlocal and node.col_offset == root_col_offset:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	497 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	498 'stdlib import "%s" follows local import: %s',
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	499 name,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	500 seenlocal,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	501 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	502
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	503 if not stdlib:
28330 f3fb24e36d61 import-checker: report local with stdlib late warning timeless <timeless@mozdev.org> parents: 27621 diff changeset	504 seenlocal = name
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	505
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	506 # Import of sibling modules should use relative imports.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	507 topname = name.split('.')[0]
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	508 if topname == topmodule:
26955 c4114e335b49 import-checker: extract function to generate a formatted warning Yuya Nishihara <yuya@tcha.org> parents: 26954 diff changeset	509 yield msg('import should be relative: %s', name)
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	510
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	511 if name in requirealias and asname != requirealias[name]:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	512 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	513 '%s module must be "as" aliased to %s',
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	514 name,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	515 requirealias[name],
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	516 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	517
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	518 elif isinstance(node, ast.ImportFrom):
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	519 # Resolve the full imported module name.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	520 if node.level > 0:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	521 fullname = '.'.join(module.split('.')[: -node.level])
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	522 if node.module:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	523 fullname += '.%s' % node.module
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	524 else:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	525 assert node.module
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	526 fullname = node.module
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	527
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	528 topname = fullname.split('.')[0]
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	529 if topname == topmodule:
26955 c4114e335b49 import-checker: extract function to generate a formatted warning Yuya Nishihara <yuya@tcha.org> parents: 26954 diff changeset	530 yield msg('import should be relative: %s', fullname)
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	531
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	532 # __future__ is special since it needs to come first and use
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	533 # symbol import.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	534 if fullname != '__future__':
34038 bc2535238de2 import-checker: allow relative import a module being checked Jun Wu <quark@fb.com> parents: 33920 diff changeset	535 if not fullname or (
bc2535238de2 import-checker: allow relative import a module being checked Jun Wu <quark@fb.com> parents: 33920 diff changeset	536 fullname in stdlib_modules
bc2535238de2 import-checker: allow relative import a module being checked Jun Wu <quark@fb.com> parents: 33920 diff changeset	537 and fullname not in localmods
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	538 and fullname + '.__init__' not in localmods
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	539 ):
26955 c4114e335b49 import-checker: extract function to generate a formatted warning Yuya Nishihara <yuya@tcha.org> parents: 26954 diff changeset	540 yield msg('relative import of stdlib module')
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	541 else:
28330 f3fb24e36d61 import-checker: report local with stdlib late warning timeless <timeless@mozdev.org> parents: 27621 diff changeset	542 seenlocal = fullname
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	543
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	544 # Direct symbol import is only allowed from certain modules and
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	545 # must occur before non-symbol imports.
29207 a09098c61fea import-checker: always build a list of imported symbols Yuya Nishihara <yuya@tcha.org> parents: 29122 diff changeset	546 found = fromlocal(node.module, node.level)
a09098c61fea import-checker: always build a list of imported symbols Yuya Nishihara <yuya@tcha.org> parents: 29122 diff changeset	547 if found and found[2]: # node.module is a package
a09098c61fea import-checker: always build a list of imported symbols Yuya Nishihara <yuya@tcha.org> parents: 29122 diff changeset	548 prefix = found[0] + '.'
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	549 symbols = (
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	550 n.name for n in node.names if not fromlocal(prefix + n.name)
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	551 )
29207 a09098c61fea import-checker: always build a list of imported symbols Yuya Nishihara <yuya@tcha.org> parents: 29122 diff changeset	552 else:
32419 d02888308235 import-checker: add a way to directly import certain symbols Siddharth Agarwal <sid0@fb.com> parents: 32374 diff changeset	553 symbols = (n.name for n in node.names)
d02888308235 import-checker: add a way to directly import certain symbols Siddharth Agarwal <sid0@fb.com> parents: 32374 diff changeset	554 symbols = [sym for sym in symbols if sym not in directsymbols]
26965 1fa66d3ad28d import-checker: reset context to verify convention in function scope Yuya Nishihara <yuya@tcha.org> parents: 26964 diff changeset	555 if node.module and node.col_offset == root_col_offset:
27272 69308357ecd1 import-checker: allow absolute imports of sub modules from local packages Yuya Nishihara <yuya@tcha.org> parents: 27018 diff changeset	556 if symbols and fullname not in allowsymbolimports:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	557 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	558 'direct symbol import %s from %s',
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	559 ', '.join(symbols),
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	560 fullname,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	561 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	562
29208 cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules Yuya Nishihara <yuya@tcha.org> parents: 29207 diff changeset	563 if symbols and seennonsymbollocal:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	564 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	565 'symbol import follows non-symbol import: %s', fullname
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	566 )
29208 cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules Yuya Nishihara <yuya@tcha.org> parents: 29207 diff changeset	567 if not symbols and fullname not in stdlib_modules:
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules Yuya Nishihara <yuya@tcha.org> parents: 29207 diff changeset	568 seennonsymbollocal = True
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	569
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	570 if not node.module:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	571 assert node.level
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	572
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	573 # Only allow 1 group per level.
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	574 if (
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	575 node.level in seenlevels
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	576 and node.col_offset == root_col_offset
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	577 ):
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	578 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	579 'multiple "from %s import" statements', '.' * node.level
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	580 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	581
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	582 # Higher-level groups come before lower-level groups.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	583 if any(node.level > l for l in seenlevels):
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	584 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	585 'higher-level import should come first: %s', fullname
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	586 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	587
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	588 seenlevels.add(node.level)
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	589
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	590 # Entries in "from .X import ( ... )" lists must be lexically
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	591 # sorted.
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	592 lastentryname = None
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	593
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	594 for n in node.names:
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	595 if lastentryname and n.name < lastentryname:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	596 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	597 'imports from %s not lexically sorted: %s < %s',
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	598 fullname,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	599 n.name,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	600 lastentryname,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	601 )
25703 1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	602
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	603 lastentryname = n.name
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	604
1a6a117d0b95 import-checker: establish modern import convention Gregory Szorc <gregory.szorc@gmail.com> parents: 25702 diff changeset	605 if n.name in requirealias and n.asname != requirealias[n.name]:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	606 yield msg(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	607 '%s from %s must be "as" aliased to %s',
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	608 n.name,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	609 fullname,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	610 requirealias[n.name],
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	611 )
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	612
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	613
ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	614 def verify_stdlib_on_own_line(root):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	615 """Given some python source, verify that stdlib imports are done
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	616 in separate statements from relative local module imports.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	617
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	618 >>> list(verify_stdlib_on_own_line(ast.parse('import sys, foo')))
26956 4b56214ebb7a import-checker: include lineno in warning message Yuya Nishihara <yuya@tcha.org> parents: 26955 diff changeset	619 [('mixed imports\\n stdlib: sys\\n relative: foo', 1)]
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	620 >>> list(verify_stdlib_on_own_line(ast.parse('import sys, os')))
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	621 []
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	622 >>> list(verify_stdlib_on_own_line(ast.parse('import foo, bar')))
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	623 []
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	624 """
25702 ab2c5163900e import-checker: establish new function for verifying import conventions Gregory Szorc <gregory.szorc@gmail.com> parents: 25701 diff changeset	625 for node in ast.walk(root):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	626 if isinstance(node, ast.Import):
20386 a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	627 from_stdlib = {False: [], True: []}
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	628 for n in node.names:
20386 a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	629 from_stdlib[n.name in stdlib_modules].append(n.name)
a05d31b040d7 import-checker: show stdlib and relative imports separately Mads Kiilerich <madski@unity3d.com> parents: 20383 diff changeset	630 if from_stdlib[True] and from_stdlib[False]:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	631 yield (
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	632 'mixed imports\n stdlib: %s\n relative: %s'
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	633 % (
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	634 ', '.join(sorted(from_stdlib[True])),
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	635 ', '.join(sorted(from_stdlib[False])),
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	636 ),
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	637 node.lineno,
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	638 )
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	639
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	640
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	641 class CircularImport(Exception):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	642 pass
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	643
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	644
24490 fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	645 def checkmod(mod, imports):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	646 shortest = {}
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	647 visit = [[mod]]
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	648 while visit:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	649 path = visit.pop(0)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	650 for i in sorted(imports.get(path[-1], [])):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	651 if len(path) < shortest.get(i, 1000):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	652 shortest[i] = len(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	653 if i in path:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	654 if i == path[0]:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	655 raise CircularImport(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	656 continue
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	657 visit.append(path + [i])
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	658
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	659
20038 c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	660 def rotatecycle(cycle):
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	661 """arrange a cycle so that the lexicographically first module listed first
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	662
24488 4b3fc46097f7 import-checker: drop duplicate element from cycle Matt Mackall <mpm@selenic.com> parents: 24487 diff changeset	663 >>> rotatecycle(['foo', 'bar'])
20038 c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	664 ['bar', 'foo', 'bar']
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	665 """
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	666 lowest = min(cycle)
c65a6937b828 import-checker: try a little harder to show fewer cycles Augie Fackler <raf@durin42.com> parents: 20037 diff changeset	667 idx = cycle.index(lowest)
24488 4b3fc46097f7 import-checker: drop duplicate element from cycle Matt Mackall <mpm@selenic.com> parents: 24487 diff changeset	668 return cycle[idx:] + cycle[:idx] + [lowest]
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	669
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	670
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	671 def find_cycles(imports):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	672 """Find cycles in an already-loaded import graph.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	673
25175 10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	674 All module names recorded in `imports` should be absolute one.
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	675
28703 a274c4f9087a py3: use print_function in import-checker timeless <timeless@mozdev.org> parents: 28702 diff changeset	676 >>> from __future__ import print_function
25175 10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	677 >>> imports = {'top.foo': ['top.bar', 'os.path', 'top.qux'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	678 ... 'top.bar': ['top.baz', 'sys'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	679 ... 'top.baz': ['top.foo'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25174 diff changeset	680 ... 'top.qux': ['top.foo']}
28703 a274c4f9087a py3: use print_function in import-checker timeless <timeless@mozdev.org> parents: 28702 diff changeset	681 >>> print('\\n'.join(sorted(find_cycles(imports))))
24487 642d245ff537 import-checker: fix rotatecycle Matt Mackall <mpm@selenic.com> parents: 22975 diff changeset	682 top.bar -> top.baz -> top.foo -> top.bar
642d245ff537 import-checker: fix rotatecycle Matt Mackall <mpm@selenic.com> parents: 22975 diff changeset	683 top.foo -> top.qux -> top.foo
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	684 """
24491 784b278b349c import-checker: rotatecycle is actually the canonical cycle key Matt Mackall <mpm@selenic.com> parents: 24490 diff changeset	685 cycles = set()
28704 1fa6fdb72275 py3: handle iter/iterkeys+iteritems python3 divergence in import-checker timeless <timeless@mozdev.org> parents: 28703 diff changeset	686 for mod in sorted(imports.keys()):
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	687 try:
24490 fb4639d5268e import-checker: make search algorithm non-recursive breadth-first Matt Mackall <mpm@selenic.com> parents: 24489 diff changeset	688 checkmod(mod, imports)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25175 diff changeset	689 except CircularImport as e:
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	690 cycle = e.args[0]
24491 784b278b349c import-checker: rotatecycle is actually the canonical cycle key Matt Mackall <mpm@selenic.com> parents: 24490 diff changeset	691 cycles.add(" -> ".join(rotatecycle(cycle)))
784b278b349c import-checker: rotatecycle is actually the canonical cycle key Matt Mackall <mpm@selenic.com> parents: 24490 diff changeset	692 return cycles
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	693
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	694
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	695 def _cycle_sortkey(c):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	696 return len(c), c
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	697
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	698
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	699 def embedded(f, modname, src):
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	700 """Extract embedded python code
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	701
33877 bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	702 >>> def _forcestr(thing):
bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	703 ... if not isinstance(thing, str):
bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	704 ... return thing.decode('ascii')
bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	705 ... return thing
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	706 >>> def test(fn, lines):
33877 bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	707 ... for s, m, f, l in embedded(fn, b"example", lines):
bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	708 ... print("%s %s %d" % (_forcestr(m), _forcestr(f), l))
bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	709 ... print(repr(_forcestr(s)))
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	710 >>> lines = [
42388 04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	711 ... 'comment',
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	712 ... ' >>> from __future__ import print_function',
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	713 ... " >>> ' multiline",
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	714 ... " ... string'",
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	715 ... ' ',
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	716 ... 'comment',
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	717 ... ' $ cat > foo.py <<EOF',
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	718 ... ' > from __future__ import print_function',
04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	719 ... ' > EOF',
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	720 ... ]
33877 bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	721 >>> test(b"example.t", lines)
40095 7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	722 example[2] doctest.py 1
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	723 "from __future__ import print_function\\n' multiline\\nstring'\\n\\n"
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	724 example[8] foo.py 7
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	725 'from __future__ import print_function\\n'
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	726 """
40095 7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	727 errors = []
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	728 for name, starts, ends, code in testparseutil.pyembedded(f, src, errors):
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	729 if not name:
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	730 # use 'doctest.py', in order to make already existing
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	731 # doctest above pass instantly
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	732 name = 'doctest.py'
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	733 # "starts" is "line number" (1-origin), but embedded() is
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	734 # expected to return "line offset" (0-origin). Therefore, this
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	735 # yields "starts - 1".
40480 99d5424eedc8 contrib: fix import-checker to not b'' module names on Python 3 Augie Fackler <augie@google.com> parents: 40095 diff changeset	736 if not isinstance(modname, str):
99d5424eedc8 contrib: fix import-checker to not b'' module names on Python 3 Augie Fackler <augie@google.com> parents: 40095 diff changeset	737 modname = modname.decode('utf8')
40095 7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 39329 diff changeset	738 yield code, "%s[%d]" % (modname, starts), name, starts - 1
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	739
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	740
28919 a94f34306bb9 import-checker: refactor source reading timeless <timeless@mozdev.org> parents: 28713 diff changeset	741 def sources(f, modname):
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	742 """Yields possibly multiple sources from a filepath
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	743
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	744 input: filepath, modulename
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	745 yields: script(string), modulename, filepath, linenumber
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	746
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	747 For embedded scripts, the modulename and filepath will be different
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	748 from the function arguments. linenumber is an offset relative to
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	749 the input file.
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	750 """
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	751 py = False
29234 393aef802535 tests: enable import checker for all python files (including no .py files) Yuya Nishihara <yuya@tcha.org> parents: 29211 diff changeset	752 if not f.endswith('.t'):
33877 bcf53149ebce contrib: make import checker always think in terms of bytes Augie Fackler <raf@durin42.com> parents: 33876 diff changeset	753 with open(f, 'rb') as src:
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	754 yield src.read(), modname, f, 0
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	755 py = True
4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	756 if py or f.endswith('.t'):
42388 04eb3c5607af contrib: fix import-checker to operate on str instead of bytes Augie Fackler <augie@google.com> parents: 41161 diff changeset	757 with open(f, 'r') as src:
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	758 for script, modname, t, line in embedded(f, modname, src):
40481 90517fad4293 contrib: tweak import-checker to always use bytes for module names Augie Fackler <augie@google.com> parents: 40480 diff changeset	759 yield script, modname.encode('utf8'), t, line
28919 a94f34306bb9 import-checker: refactor source reading timeless <timeless@mozdev.org> parents: 28713 diff changeset	760
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	761
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	762 def main(argv):
25063 723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	763 if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
28703 a274c4f9087a py3: use print_function in import-checker timeless <timeless@mozdev.org> parents: 28702 diff changeset	764 print('Usage: %s {-\|file [file] [file] ...}')
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	765 return 1
25063 723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	766 if argv[1] == '-':
723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	767 argv = argv[:1]
723e364488f4 import-checker: add xargs like mode FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24669 diff changeset	768 argv.extend(l.rstrip() for l in sys.stdin.readlines())
32508 4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	769 localmodpaths = {}
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	770 used_imports = {}
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	771 any_errors = False
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	772 for source_path in argv[1:]:
32374 194b0f781132 import-checker: drop workaround for pure modules Yuya Nishihara <yuya@tcha.org> parents: 32372 diff changeset	773 modname = dotted_name_of_path(source_path)
32508 4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	774 localmodpaths[modname] = source_path
32509 a025ec43856c import-checker: guess names of C extension modules Yuya Nishihara <yuya@tcha.org> parents: 32508 diff changeset	775 localmods = populateextmods(localmodpaths)
32508 4c712b90c60a import-checker: convert localmods to a set of module names Yuya Nishihara <yuya@tcha.org> parents: 32507 diff changeset	776 for localmodname, source_path in sorted(localmodpaths.items()):
33890 3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	777 if not isinstance(localmodname, bytes):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	778 # This is only safe because all hg's files are ascii
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names Augie Fackler <raf@durin42.com> parents: 33877 diff changeset	779 localmodname = localmodname.encode('ascii')
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	780 for src, modname, name, line in sources(source_path, localmodname):
28920 cdf331b54eb8 import-checker: track SyntaxErrors timeless <timeless@mozdev.org> parents: 28919 diff changeset	781 try:
cdf331b54eb8 import-checker: track SyntaxErrors timeless <timeless@mozdev.org> parents: 28919 diff changeset	782 used_imports[modname] = sorted(
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	783 imported_modules(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	784 src, modname, name, localmods, ignore_nested=True
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	785 )
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	786 )
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	787 for error, lineno in verify_import_convention(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	788 modname, src, localmods
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	789 ):
28920 cdf331b54eb8 import-checker: track SyntaxErrors timeless <timeless@mozdev.org> parents: 28919 diff changeset	790 any_errors = True
28922 4ec62a084e5c import-checker: parse python code from .t files timeless <timeless@mozdev.org> parents: 28921 diff changeset	791 print('%s:%d: %s' % (source_path, lineno + line, error))
28920 cdf331b54eb8 import-checker: track SyntaxErrors timeless <timeless@mozdev.org> parents: 28919 diff changeset	792 except SyntaxError as e:
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	793 print(
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	794 '%s:%d: SyntaxError: %s' % (source_path, e.lineno + line, e)
57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	795 )
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	796 cycles = find_cycles(used_imports)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	797 if cycles:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	798 firstmods = set()
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	799 for c in sorted(cycles, key=_cycle_sortkey):
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	800 first = c.split()[0]
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	801 # As a rough cut, ignore any cycle that starts with the
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	802 # same module as some other cycle. Otherwise we see lots
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	803 # of cycles that are effectively duplicates.
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	804 if first in firstmods:
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	805 continue
28703 a274c4f9087a py3: use print_function in import-checker timeless <timeless@mozdev.org> parents: 28702 diff changeset	806 print('Import cycle:', c)
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	807 firstmods.add(first)
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	808 any_errors = True
25731 cd1daab5d036 import-checker.py: exit with code 0 if no error is detected FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 25703 diff changeset	809 return any_errors != 0
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	810
43075 57875cf423c9 style: run a patched black on a subset of mercurial Augie Fackler <augie@google.com> parents: 42813 diff changeset	811
20036 e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	812 if __name__ == '__main__':
e5d51fa51aba contrib: add an import checker Augie Fackler <raf@durin42.com> parents: diff changeset	813 sys.exit(int(main(sys.argv)))

Mercurial > public > mercurial-scm > hg

annotate contrib/import-checker.py @ 43084:c2e284cee333