mercurial-scm/hg-stable: contrib/automation/hgautomation/linux.py annotate

annotate contrib/automation/hgautomation/linux.py @ 45685:57b5452a55d5

pyoxidizer: produce working Python 3 Windows installers (issue6366) While we've had code to produce Python 3 Windows installers with PyOxidizer, we haven't been advertising them on the web site due to a bug in making TLS connections and issues around resource handling. This commit upgrades our PyOxidizer install and configuration to use a recent Git commit of PyOxidizer. This new version of PyOxidizer contains a *ton* of changes, improvements, and bug fixes. Notably, Windows shared distributions now mostly "just work" and the TLS bug and random problems with Python extension modules in the standard library go away. And Python has been upgraded from 3.7 to 3.8.6. The price we pay for this upgrade is a ton of backwards incompatible changes to Starlark. I applied this commit (the overall series actually) on stable to produce Windows installers for Mercurial 5.5.2, which I published shortly before submitting this commit for review. In order to get the stable branch working, I decided to take a less aggressive approach to Python resource management. Previously, we were attempting to load all Python modules from memory and were performing some hacks to copy Mercurial's non-module resources into additional directories in Starlark. This commit implements a resource callback function in Starlark (a new feature since PyOxidizer 0.7) to dynamically assign standard library resources to in-memory loading and all other resources to filesystem loading. This means that Mercurial's files and all the other packages we ship in the Windows installers (e.g. certifi and pygments) are loaded from the filesystem instead of from memory. This avoids issues due to lack of __file__ and enables us to ship a working Python 3 installer on Windows. The end state of the install layout after this patch is not ideal for @: we still copy resource files like templates and help text to directories next to the hg.exe executable. There is code in @ to use importlib.resources to load these files and we could likely remove these copies once this lands on @. But for now, the install layout mimics what we've shipped for seemingly forever and is backwards compatible. It allows us to achieve the milestone of working Python 3 Windows installers and gets us a giant step closer to deleting Python 2. Differential Revision: https://phab.mercurial-scm.org/D9148

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Sun, 04 Oct 2020 22:32:41 -0700
parents	6a36e2d2011f
children	7a907388a4a5

rev	line source
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # linux.py - Linux specific automation functionality
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 # no-check-code because Python 3 native.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10 import os
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11 import pathlib
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12 import shlex
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13 import subprocess
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 import tempfile
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	16 from .ssh import exec_command
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 # Linux distributions that are supported.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	20 DISTROS = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	21 'debian9',
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	22 'debian10',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	23 'ubuntu18.04',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	24 'ubuntu19.04',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	25 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	26
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	27 INSTALL_PYTHONS = r'''
43315 388ada1c07c6 automation: install Python 2.7.17, 3.7.5, and PyPy 7.2.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 43235 diff changeset	28 PYENV2_VERSIONS="2.7.17 pypy2.7-7.2.0"
45644 56281b75264a contrib: install latest versions of Python Gregory Szorc <gregory.szorc@gmail.com> parents: 44745 diff changeset	29 PYENV3_VERSIONS="3.5.10 3.6.12 3.7.9 3.8.6 pypy3.5-7.0.0 pypy3.6-7.3.0"
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 git clone https://github.com/pyenv/pyenv.git /hgdev/pyenv
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 pushd /hgdev/pyenv
45644 56281b75264a contrib: install latest versions of Python Gregory Szorc <gregory.szorc@gmail.com> parents: 44745 diff changeset	33 git checkout 8bf79eb44ce1ea0a8b36544eb3e8a694a3a6ed78
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	34 popd
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	35
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36 export PYENV_ROOT="/hgdev/pyenv"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	37 export PATH="$PYENV_ROOT/bin:$PATH"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	38
43016 cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	39 # pip 19.2.3.
cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	40 PIP_SHA256=57e3643ff19f018f8a00dfaa6b7e4620e3c1a7a2171fd218425366ec006b3bfe
cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	41 wget -O get-pip.py --progress dot:mega https://github.com/pypa/get-pip/raw/309a56c5fd94bd1134053a541cb4657a4e47e09d/get-pip.py
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	42 echo "${PIP_SHA256} get-pip.py" \| sha256sum --check -
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	43
43016 cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	44 VIRTUALENV_SHA256=f78d81b62d3147396ac33fc9d77579ddc42cc2a98dd9ea38886f616b33bc7fb2
cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	45 VIRTUALENV_TARBALL=virtualenv-16.7.5.tar.gz
cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	46 wget -O ${VIRTUALENV_TARBALL} --progress dot:mega https://files.pythonhosted.org/packages/66/f0/6867af06d2e2f511e4e1d7094ff663acdebc4f15d4a0cb0fed1007395124/${VIRTUALENV_TARBALL}
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	47 echo "${VIRTUALENV_SHA256} ${VIRTUALENV_TARBALL}" \| sha256sum --check -
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	48
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	49 for v in ${PYENV2_VERSIONS}; do
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	50 pyenv install -v ${v}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	51 ${PYENV_ROOT}/versions/${v}/bin/python get-pip.py
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	52 ${PYENV_ROOT}/versions/${v}/bin/pip install ${VIRTUALENV_TARBALL}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	53 ${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/requirements-py2.txt
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	54 done
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	55
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	56 for v in ${PYENV3_VERSIONS}; do
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	57 pyenv install -v ${v}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	58 ${PYENV_ROOT}/versions/${v}/bin/python get-pip.py
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	59 ${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/requirements-py3.txt
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	60 done
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	61
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	62 pyenv global ${PYENV2_VERSIONS} ${PYENV3_VERSIONS} system
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	63 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	64 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	65 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	66
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67
42718 89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	68 INSTALL_RUST = r'''
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	69 RUSTUP_INIT_SHA256=a46fe67199b7bcbbde2dcbc23ae08db6f29883e260e23899a88b9073effc9076
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	70 wget -O rustup-init --progress dot:mega https://static.rust-lang.org/rustup/archive/1.18.3/x86_64-unknown-linux-gnu/rustup-init
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	71 echo "${RUSTUP_INIT_SHA256} rustup-init" \| sha256sum --check -
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	72
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	73 chmod +x rustup-init
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	74 sudo -H -u hg -g hg ./rustup-init -y
45645 6a36e2d2011f contrib: install Rust 1.46.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 45644 diff changeset	75 sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup install 1.31.1 1.46.0
42718 89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	76 sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup component add clippy
44745 6a6c15cea1fa contrib: install PyOxidizer in Linux and Windows environments Gregory Szorc <gregory.szorc@gmail.com> parents: 44736 diff changeset	77
45685 57b5452a55d5 pyoxidizer: produce working Python 3 Windows installers (issue6366) Gregory Szorc <gregory.szorc@gmail.com> parents: 45645 diff changeset	78 sudo -H -u hg -g hg /home/hg/.cargo/bin/cargo install --git https://github.com/indygreg/PyOxidizer.git --rev 4697fb25918dfad6dc73288daeea501063963a08 pyoxidizer
42718 89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	79 '''
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	80
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	81
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	82 BOOTSTRAP_VIRTUALENV = r'''
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	83 /usr/bin/virtualenv /hgdev/venv-bootstrap
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	84
43016 cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	85 HG_SHA256=35fc8ba5e0379c1b3affa2757e83fb0509e8ac314cbd9f1fd133cf265d16e49f
cbd94ee3a72e automation: upgrade packages in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	86 HG_TARBALL=mercurial-5.1.1.tar.gz
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 wget -O ${HG_TARBALL} --progress dot:mega https://www.mercurial-scm.org/release/${HG_TARBALL}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 echo "${HG_SHA256} ${HG_TARBALL}" \| sha256sum --check -
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 /hgdev/venv-bootstrap/bin/pip install ${HG_TARBALL}
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	92 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	93 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	94 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	97 BOOTSTRAP_DEBIAN = (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	98 r'''
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 #!/bin/bash
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 set -ex
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 DISTRO=`grep DISTRIB_ID /etc/lsb-release \| awk -F= '{{print $2}}'`
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 DEBIAN_VERSION=`cat /etc/debian_version`
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 LSB_RELEASE=`lsb_release -cs`
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107 sudo /usr/sbin/groupadd hg
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 sudo /usr/sbin/groupadd docker
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109 sudo /usr/sbin/useradd -g hg -G sudo,docker -d /home/hg -m -s /bin/bash hg
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110 sudo mkdir /home/hg/.ssh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 sudo cp ~/.ssh/authorized_keys /home/hg/.ssh/authorized_keys
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 sudo chown -R hg:hg /home/hg/.ssh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 sudo chmod 700 /home/hg/.ssh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 sudo chmod 600 /home/hg/.ssh/authorized_keys
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 cat << EOF \| sudo tee /etc/sudoers.d/90-hg
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117 hg ALL=(ALL) NOPASSWD:ALL
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118 EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 sudo apt-get update
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121 sudo DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 # Install packages necessary to set up Docker Apt repo.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 sudo DEBIAN_FRONTEND=noninteractive apt-get -yq install --no-install-recommends \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125 apt-transport-https \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	126 gnupg
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 cat > docker-apt-key << EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 -----BEGIN PGP PUBLIC KEY BLOCK-----
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131 mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132 lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133 38UuLa+z077PxyxQhu5BbqntTPQMfiyqEiU+BKbq2WmANUKQf+1AmZY/IruOXbnq
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 L4C1+gJ8vfmXQt99npCaxEjaNRVYfOS8QcixNzHUYnb6emjlANyEVlZzeqo7XKl7
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 UrwV5inawTSzWNvtjEjj4nJL8NsLwscpLPQUhTQ+7BbQXAwAmeHCUTQIvvWXqw0N
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136 cmhh4HgeQscQHYgOJjjDVfoY5MucvglbIgCqfzAHW9jxmRL4qbMZj+b1XoePEtht
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137 ku4bIQN1X5P07fNWzlgaRL5Z4POXDDZTlIQ/El58j9kp4bnWRCJW0lya+f8ocodo
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 vZZ+Doi+fy4D5ZGrL4XEcIQP/Lv5uFyf+kQtl/94VFYVJOleAv8W92KdgDkhTcTD
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 G7c0tIkVEKNUq48b3aQ64NOZQW7fVjfoKwEZdOqPE72Pa45jrZzvUFxSpdiNk2tZ
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	140 XYukHjlxxEgBdC/J3cMMNRE1F4NCA3ApfV1Y7/hTeOnmDuDYwr9/obA8t016Yljj
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141 q5rdkywPf4JF8mXUW5eCN1vAFHxeg9ZWemhBtQmGxXnw9M+z6hWwc6ahmwARAQAB
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	142 tCtEb2NrZXIgUmVsZWFzZSAoQ0UgZGViKSA8ZG9ja2VyQGRvY2tlci5jb20+iQI3
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 BBMBCgAhBQJYrefAAhsvBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJEI2BgDwO
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 v82IsskP/iQZo68flDQmNvn8X5XTd6RRaUH33kXYXquT6NkHJciS7E2gTJmqvMqd
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145 tI4mNYHCSEYxI5qrcYV5YqX9P6+Ko+vozo4nseUQLPH/ATQ4qL0Zok+1jkag3Lgk
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 jonyUf9bwtWxFp05HC3GMHPhhcUSexCxQLQvnFWXD2sWLKivHp2fT8QbRGeZ+d3m
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147 6fqcd5Fu7pxsqm0EUDK5NL+nPIgYhN+auTrhgzhK1CShfGccM/wfRlei9Utz6p9P
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148 XRKIlWnXtT4qNGZNTN0tR+NLG/6Bqd8OYBaFAUcue/w1VW6JQ2VGYZHnZu9S8LMc
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	149 FYBa5Ig9PxwGQOgq6RDKDbV+PqTQT5EFMeR1mrjckk4DQJjbxeMZbiNMG5kGECA8
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150 g383P3elhn03WGbEEa4MNc3Z4+7c236QI3xWJfNPdUbXRaAwhy/6rTSFbzwKB0Jm
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151 ebwzQfwjQY6f55MiI/RqDCyuPj3r3jyVRkK86pQKBAJwFHyqj9KaKXMZjfVnowLh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152 9svIGfNbGHpucATqREvUHuQbNnqkCx8VVhtYkhDb9fEP2xBu5VvHbR+3nfVhMut5
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	153 G34Ct5RS7Jt6LIfFdtcn8CaSas/l1HbiGeRgc70X/9aYx/V/CEJv0lIe8gP6uDoW
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	154 FPIZ7d6vH+Vro6xuWEGiuMaiznap2KhZmpkgfupyFmplh0s6knymuQINBFit2ioB
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	155 EADneL9S9m4vhU3blaRjVUUyJ7b/qTjcSylvCH5XUE6R2k+ckEZjfAMZPLpO+/tF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	156 M2JIJMD4SifKuS3xck9KtZGCufGmcwiLQRzeHF7vJUKrLD5RTkNi23ydvWZgPjtx
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	157 Q+DTT1Zcn7BrQFY6FgnRoUVIxwtdw1bMY/89rsFgS5wwuMESd3Q2RYgb7EOFOpnu
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	158 w6da7WakWf4IhnF5nsNYGDVaIHzpiqCl+uTbf1epCjrOlIzkZ3Z3Yk5CM/TiFzPk
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	159 z2lLz89cpD8U+NtCsfagWWfjd2U3jDapgH+7nQnCEWpROtzaKHG6lA3pXdix5zG8
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160 eRc6/0IbUSWvfjKxLLPfNeCS2pCL3IeEI5nothEEYdQH6szpLog79xB9dVnJyKJb
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	161 VfxXnseoYqVrRz2VVbUI5Blwm6B40E3eGVfUQWiux54DspyVMMk41Mx7QJ3iynIa
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	162 1N4ZAqVMAEruyXTRTxc9XW0tYhDMA/1GYvz0EmFpm8LzTHA6sFVtPm/ZlNCX6P1X
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	163 zJwrv7DSQKD6GGlBQUX+OeEJ8tTkkf8QTJSPUdh8P8YxDFS5EOGAvhhpMBYD42kQ
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164 pqXjEC+XcycTvGI7impgv9PDY1RCC1zkBjKPa120rNhv/hkVk/YhuGoajoHyy4h7
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	165 ZQopdcMtpN2dgmhEegny9JCSwxfQmQ0zK0g7m6SHiKMwjwARAQABiQQ+BBgBCAAJ
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	166 BQJYrdoqAhsCAikJEI2BgDwOv82IwV0gBBkBCAAGBQJYrdoqAAoJEH6gqcPyc/zY
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	167 1WAP/2wJ+R0gE6qsce3rjaIz58PJmc8goKrir5hnElWhPgbq7cYIsW5qiFyLhkdp
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168 YcMmhD9mRiPpQn6Ya2w3e3B8zfIVKipbMBnke/ytZ9M7qHmDCcjoiSmwEXN3wKYI
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169 mD9VHONsl/CG1rU9Isw1jtB5g1YxuBA7M/m36XN6x2u+NtNMDB9P56yc4gfsZVES
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170 KA9v+yY2/l45L8d/WUkUi0YXomn6hyBGI7JrBLq0CX37GEYP6O9rrKipfz73XfO7
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171 JIGzOKZlljb/D9RX/g7nRbCn+3EtH7xnk+TK/50euEKw8SMUg147sJTcpQmv6UzZ
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172 cM4JgL0HbHVCojV4C/plELwMddALOFeYQzTif6sMRPf+3DSj8frbInjChC3yOLy0
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	173 6br92KFom17EIj2CAcoeq7UPhi2oouYBwPxh5ytdehJkoo+sN7RIWua6P2WSmon5
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	174 U888cSylXC0+ADFdgLX9K2zrDVYUG1vo8CX0vzxFBaHwN6Px26fhIT1/hYUHQR1z
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	175 VfNDcyQmXqkOnZvvoMfz/Q0s9BhFJ/zU6AgQbIZE/hm1spsfgvtsD1frZfygXJ9f
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176 irP+MSAI80xHSf91qSRZOj4Pl3ZJNbq4yYxv0b1pkMqeGdjdCYhLU+LZ4wbQmpCk
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	177 SVe2prlLureigXtmZfkqevRz7FrIZiu9ky8wnCAPwC7/zmS18rgP/17bOtL4/iIz
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	178 QhxAAoAMWVrGyJivSkjhSGx1uCojsWfsTAm11P7jsruIL61ZzMUVE2aM3Pmj5G+W
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	179 9AcZ58Em+1WsVnAXdUR//bMmhyr8wL/G1YO1V3JEJTRdxsSxdYa4deGBBY/Adpsw
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	180 24jxhOJR+lsJpqIUeb999+R8euDhRHG9eFO7DRu6weatUJ6suupoDTRWtr/4yGqe
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	181 dKxV3qQhNLSnaAzqW/1nA3iUB4k7kCaKZxhdhDbClf9P37qaRW467BLCVO/coL3y
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182 Vm50dwdrNtKpMBh3ZpbB1uJvgi9mXtyBOMJ3v8RZeDzFiG8HdCtg9RvIt/AIFoHR
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	183 H3S+U79NT6i0KPzLImDfs8T7RlpyuMc4Ufs8ggyg9v3Ae6cN3eQyxcK3w0cbBwsh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	184 /nQNfsA6uu+9H7NhbehBMhYnpNZyrHzCmzyXkauwRAqoCbGCNykTRwsur9gS41TQ
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	185 M8ssD1jFheOJf3hODnkKU+HKjvMROl1DK7zdmLdNzA1cvtZH/nCC9KPj1z8QC47S
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	186 xx+dTZSx4ONAhwbS/LN3PoKtn8LPjY9NP9uDWI+TWYquS2U+KHDrBDlsgozDbs/O
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	187 jCxcpDzNmXpWQHEtHU7649OXHP7UeNST1mCUCH5qdank0V1iejF6/CfTFU4MfcrG
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	188 YT90qFF93M3v01BbxP+EIY2/9tiIPbrd
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	189 =0YYh
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	190 -----END PGP PUBLIC KEY BLOCK-----
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	191 EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	192
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	193 sudo apt-key add docker-apt-key
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	194
43013 198b51d453fe automation: use LSB_RELEASE instead of DEBIAN_VERSION Gregory Szorc <gregory.szorc@gmail.com> parents: 42718 diff changeset	195 if [ "$LSB_RELEASE" = "stretch" ]; then
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	196 cat << EOF \| sudo tee -a /etc/apt/sources.list
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	197 # Need backports for clang-format-6.0
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	198 deb http://deb.debian.org/debian stretch-backports main
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	199 EOF
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	200 fi
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	202 if [ "$LSB_RELEASE" = "stretch" -o "$LSB_RELEASE" = "buster" ]; then
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	203 cat << EOF \| sudo tee -a /etc/apt/sources.list
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204 # Sources are useful if we want to compile things locally.
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	205 deb-src http://deb.debian.org/debian $LSB_RELEASE main
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	206 deb-src http://security.debian.org/debian-security $LSB_RELEASE/updates main
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	207 deb-src http://deb.debian.org/debian $LSB_RELEASE-updates main
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	208 deb-src http://deb.debian.org/debian $LSB_RELEASE-backports main
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	210 deb [arch=amd64] https://download.docker.com/linux/debian $LSB_RELEASE stable
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213 elif [ "$DISTRO" = "Ubuntu" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214 cat << EOF \| sudo tee -a /etc/apt/sources.list
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	215 deb [arch=amd64] https://download.docker.com/linux/ubuntu $LSB_RELEASE stable
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	216 EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	217
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	218 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220 sudo apt-get update
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222 PACKAGES="\
43017 dffa9d3c2491 automation: install awscli and python3-boto3 packages Gregory Szorc <gregory.szorc@gmail.com> parents: 43016 diff changeset	223 awscli \
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	224 btrfs-progs \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	225 build-essential \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	226 bzr \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	227 clang-format-6.0 \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	228 cvs \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	229 darcs \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	230 debhelper \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	231 devscripts \
43014 136c2536d83f automation: always install docker-ce Gregory Szorc <gregory.szorc@gmail.com> parents: 43013 diff changeset	232 docker-ce \
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233 dpkg-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 dstat \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235 emacs \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236 gettext \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	237 git \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	238 htop \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	239 iotop \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	240 jfsutils \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	241 libbz2-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	242 libexpat1-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	243 libffi-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	244 libgdbm-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	245 liblzma-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	246 libncurses5-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	247 libnss3-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	248 libreadline-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	249 libsqlite3-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	250 libssl-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	251 netbase \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	252 ntfs-3g \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	253 nvme-cli \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	254 pyflakes \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	255 pyflakes3 \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	256 pylint \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257 pylint3 \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 python-all-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259 python-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	260 python-docutils \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	261 python-fuzzywuzzy \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	262 python-pygments \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	263 python-subversion \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	264 python-vcr \
43017 dffa9d3c2491 automation: install awscli and python3-boto3 packages Gregory Szorc <gregory.szorc@gmail.com> parents: 43016 diff changeset	265 python3-boto3 \
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	266 python3-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	267 python3-docutils \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	268 python3-fuzzywuzzy \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	269 python3-pygments \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270 python3-vcr \
43413 acdd4f28f6a3 automation: install python3-venv Debian package Gregory Szorc <gregory.szorc@gmail.com> parents: 43315 diff changeset	271 python3-venv \
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 rsync \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 sqlite3 \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274 subversion \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275 tcl-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 tk-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 tla \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278 unzip \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279 uuid-dev \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 vim \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 virtualenv \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 wget \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283 xfsprogs \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284 zip \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285 zlib1g-dev"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286
43013 198b51d453fe automation: use LSB_RELEASE instead of DEBIAN_VERSION Gregory Szorc <gregory.szorc@gmail.com> parents: 42718 diff changeset	287 if [ "LSB_RELEASE" = "stretch" ]; then
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288 PACKAGES="$PACKAGES linux-perf"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 elif [ "$DISTRO" = "Ubuntu" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 PACKAGES="$PACKAGES linux-tools-common"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	293 # Monotone only available in older releases.
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43017 diff changeset	294 if [ "$LSB_RELEASE" = "stretch" -o "$LSB_RELEASE" = "xenial" ]; then
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 PACKAGES="$PACKAGES monotone"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 sudo DEBIAN_FRONTEND=noninteractive apt-get -yq install --no-install-recommends $PACKAGES
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 # Create clang-format symlink so test harness finds it.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301 sudo update-alternatives --install /usr/bin/clang-format clang-format \
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 /usr/bin/clang-format-6.0 1000
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304 sudo mkdir /hgdev
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305 # Will be normalized to hg:hg later.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 sudo chown `whoami` /hgdev
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307
42718 89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	308 {install_rust}
89ba81771fc9 automation: install Rust in Linux environment Gregory Szorc <gregory.szorc@gmail.com> parents: 42716 diff changeset	309
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 cp requirements-py2.txt /hgdev/requirements-py2.txt
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311 cp requirements-py3.txt /hgdev/requirements-py3.txt
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313 # Disable the pip version check because it uses the network and can
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314 # be annoying.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315 cat << EOF \| sudo tee -a /etc/pip.conf
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	316 [global]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 disable-pip-version-check = True
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318 EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 {install_pythons}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 {bootstrap_virtualenv}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323 /hgdev/venv-bootstrap/bin/hg clone https://www.mercurial-scm.org/repo/hg /hgdev/src
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	325 # Mark the repo as non-publishing.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 cat >> /hgdev/src/.hg/hgrc << EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327 [phases]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328 publish = false
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329 EOF
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	330
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331 sudo chown -R hg:hg /hgdev
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	332 '''.lstrip()
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	333 .format(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	334 install_rust=INSTALL_RUST,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	335 install_pythons=INSTALL_PYTHONS,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	336 bootstrap_virtualenv=BOOTSTRAP_VIRTUALENV,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	337 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	338 .replace('\r\n', '\n')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	339 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	340
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	341
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342 # Prepares /hgdev for operations.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 PREPARE_HGDEV = '''
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344 #!/bin/bash
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346 set -e
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	347
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348 FS=$1
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	349
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	350 ensure_device() {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	351 if [ -z "${DEVICE}" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	352 echo "could not find block device to format"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	353 exit 1
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	354 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	355 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	356
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357 # Determine device to partition for extra filesystem.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358 # If only 1 volume is present, it will be the root volume and
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359 # should be /dev/nvme0. If multiple volumes are present, the
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360 # root volume could be nvme0 or nvme1. Use whichever one doesn't have
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	361 # a partition.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	362 if [ -e /dev/nvme1n1 ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	363 if [ -e /dev/nvme0n1p1 ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	364 DEVICE=/dev/nvme1n1
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365 else
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	366 DEVICE=/dev/nvme0n1
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	367 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	368 else
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	369 DEVICE=
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372 sudo mkdir /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	373
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	374 if [ "${FS}" != "default" -a "${FS}" != "tmpfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	375 ensure_device
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376 echo "creating ${FS} filesystem on ${DEVICE}"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379 if [ "${FS}" = "default" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380 :
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382 elif [ "${FS}" = "btrfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383 sudo mkfs.btrfs ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	386 elif [ "${FS}" = "ext3" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	387 # lazy_journal_init speeds up filesystem creation at the expense of
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	388 # integrity if things crash. We are an ephemeral instance, so we don't
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	389 # care about integrity.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	390 sudo mkfs.ext3 -E lazy_journal_init=1 ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	391 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	392
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	393 elif [ "${FS}" = "ext4" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	394 sudo mkfs.ext4 -E lazy_journal_init=1 ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	395 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397 elif [ "${FS}" = "jfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398 sudo mkfs.jfs ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	400
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	401 elif [ "${FS}" = "tmpfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	402 echo "creating tmpfs volume in /hgwork"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	403 sudo mount -t tmpfs -o size=1024M tmpfs /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	405 elif [ "${FS}" = "xfs" ]; then
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	406 sudo mkfs.xfs ${DEVICE}
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	407 sudo mount ${DEVICE} /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	408
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	409 else
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	410 echo "unsupported filesystem: ${FS}"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	411 exit 1
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	412 fi
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	413
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	414 echo "/hgwork ready"
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	415
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	416 sudo chown hg:hg /hgwork
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	417 mkdir /hgwork/tmp
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	418 chown hg:hg /hgwork/tmp
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	419
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	420 rsync -a /hgdev/src /hgwork/
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	421 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	422 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	423 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	424
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	425
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	426 HG_UPDATE_CLEAN = '''
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	427 set -ex
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	428
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	429 HG=/hgdev/venv-bootstrap/bin/hg
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431 cd /hgwork/src
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 ${HG} --config extensions.purge= purge --all
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433 ${HG} update -C $1
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434 ${HG} log -r .
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	435 '''.lstrip().replace(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	436 '\r\n', '\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	437 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440 def prepare_exec_environment(ssh_client, filesystem='default'):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441 """Prepare an EC2 instance to execute things.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	443 The AMI has an ``/hgdev`` bootstrapped with various Python installs
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	444 and a clone of the Mercurial repo.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	446 In EC2, EBS volumes launched from snapshots have wonky performance behavior.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	447 Notably, blocks have to be copied on first access, which makes volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	448 I/O extremely slow on fresh volumes.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	449
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	450 Furthermore, we may want to run operations, tests, etc on alternative
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	451 filesystems so we examine behavior on different filesystems.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	452
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	453 This function is used to facilitate executing operations on alternate
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	454 volumes.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	455 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	456 sftp = ssh_client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	457
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	458 with sftp.open('/hgdev/prepare-hgdev', 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 fh.write(PREPARE_HGDEV)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	460 fh.chmod(0o0777)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	462 command = 'sudo /hgdev/prepare-hgdev %s' % filesystem
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	463 chan, stdin, stdout = exec_command(ssh_client, command)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	464 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	465
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	466 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	467 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	468
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	469 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	470
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	471 if res:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	472 raise Exception('non-0 exit code updating working directory; %d' % res)
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	473
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	474
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	475 def synchronize_hg(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	476 source_path: pathlib.Path, ec2_instance, revision: str = None
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	477 ):
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	478 """Synchronize a local Mercurial source path to remote EC2 instance."""
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	479
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	480 with tempfile.TemporaryDirectory() as temp_dir:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	481 temp_dir = pathlib.Path(temp_dir)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	482
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	483 ssh_dir = temp_dir / '.ssh'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	484 ssh_dir.mkdir()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	485 ssh_dir.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	486
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	487 public_ip = ec2_instance.public_ip_address
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	488
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	489 ssh_config = ssh_dir / 'config'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	490
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	491 with ssh_config.open('w', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	492 fh.write('Host %s\n' % public_ip)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	493 fh.write(' User hg\n')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	494 fh.write(' StrictHostKeyChecking no\n')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	495 fh.write(' UserKnownHostsFile %s\n' % (ssh_dir / 'known_hosts'))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	496 fh.write(' IdentityFile %s\n' % ec2_instance.ssh_private_key_path)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	497
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	498 if not (source_path / '.hg').is_dir():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	499 raise Exception(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	500 '%s is not a Mercurial repository; synchronization '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	501 'not yet supported' % source_path
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	502 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	503
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	504 env = dict(os.environ)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	505 env['HGPLAIN'] = '1'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	506 env['HGENCODING'] = 'utf-8'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	507
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	508 hg_bin = source_path / 'hg'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	509
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	510 res = subprocess.run(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511 ['python2.7', str(hg_bin), 'log', '-r', revision, '-T', '{node}'],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	512 cwd=str(source_path),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	513 env=env,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	514 check=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	515 capture_output=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	516 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	517
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	518 full_revision = res.stdout.decode('ascii')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	519
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	520 args = [
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	521 'python2.7',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	522 str(hg_bin),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	523 '--config',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	524 'ui.ssh=ssh -F %s' % ssh_config,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	525 '--config',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	526 'ui.remotecmd=/hgdev/venv-bootstrap/bin/hg',
42658 9e0f1c80cddb automation: push changes affecting .hgtags Gregory Szorc <gregory.szorc@gmail.com> parents: 42651 diff changeset	527 # Also ensure .hgtags changes are present so auto version
9e0f1c80cddb automation: push changes affecting .hgtags Gregory Szorc <gregory.szorc@gmail.com> parents: 42651 diff changeset	528 # calculation works.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	529 'push',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	530 '-f',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	531 '-r',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	532 full_revision,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	533 '-r',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	534 'file(.hgtags)',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	535 'ssh://%s//hgwork/src' % public_ip,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	536 ]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	537
42651 24cd5b0ba5b3 automation: allow exit code of 1 for `hg push` Gregory Szorc <gregory.szorc@gmail.com> parents: 42312 diff changeset	538 res = subprocess.run(args, cwd=str(source_path), env=env)
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push` Gregory Szorc <gregory.szorc@gmail.com> parents: 42312 diff changeset	539
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push` Gregory Szorc <gregory.szorc@gmail.com> parents: 42312 diff changeset	540 # Allow 1 (no-op) to not trigger error.
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push` Gregory Szorc <gregory.szorc@gmail.com> parents: 42312 diff changeset	541 if res.returncode not in (0, 1):
24cd5b0ba5b3 automation: allow exit code of 1 for `hg push` Gregory Szorc <gregory.szorc@gmail.com> parents: 42312 diff changeset	542 res.check_returncode()
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544 # TODO support synchronizing dirty working directory.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546 sftp = ec2_instance.ssh_client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	547
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	548 with sftp.open('/hgdev/hgup', 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 fh.write(HG_UPDATE_CLEAN)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	552 chan, stdin, stdout = exec_command(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	553 ec2_instance.ssh_client, '/hgdev/hgup %s' % full_revision
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	554 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	555 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	556
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	557 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	558 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	559
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	560 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	561
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	562 if res:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	563 raise Exception(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	564 'non-0 exit code updating working directory; %d' % res
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	565 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	566
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	568 def run_tests(ssh_client, python_version, test_flags=None):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569 """Run tests on a remote Linux machine via an SSH client."""
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	570 test_flags = test_flags or []
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 print('running tests')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	574 if python_version == 'system2':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	575 python = '/usr/bin/python2'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	576 elif python_version == 'system3':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	577 python = '/usr/bin/python3'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	578 elif python_version.startswith('pypy'):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579 python = '/hgdev/pyenv/shims/%s' % python_version
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	580 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	581 python = '/hgdev/pyenv/shims/python%s' % python_version
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	582
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	583 test_flags = ' '.join(shlex.quote(a) for a in test_flags)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	584
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	585 command = (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586 '/bin/sh -c "export TMPDIR=/hgwork/tmp; '
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	587 'cd /hgwork/src/tests && %s run-tests.py %s"' % (python, test_flags)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	588 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	589
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	590 chan, stdin, stdout = exec_command(ssh_client, command)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	591
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	592 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	593
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	594 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	596
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	597 return chan.recv_exit_status()

Mercurial > public > mercurial-scm > hg-stable

annotate contrib/automation/hgautomation/linux.py @ 45685:57b5452a55d5