mercurial-scm/hg-stable: contrib/automation/hgautomation/aws.py annotate

annotate contrib/automation/hgautomation/aws.py @ 42312:65b3ef162b39

automation: initial support for running Linux tests Building on top of our Windows automation support, this commit implements support for performing automated tasks on remote Linux machines. Specifically, we implement support for running tests on ephemeral EC2 instances. This seems to be a worthwhile place to start, as building packages on Linux is more or less a solved problem because we already have facilities for building in Docker containers, which provide "good enough" reproducibility guarantees. The new `run-tests-linux` command works similarly to `run-tests-windows`: it ensures an AMI with hg dependencies is available, provisions a temporary EC2 instance with this AMI, pushes local changes to that instance via SSH, then invokes `run-tests.py`. Using this new command, I am able to run the entire test harness substantially faster then I am on my local machine courtesy of access to massive core EC2 instances: wall: 16:20 ./run-tests.py -l (i7-6700K) wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge ~3 minute wall time to run pretty much the entire test harness is not too bad! The AMIs install multiple versions of Python. And the run-tests-linux command specifies which one to use: automation.py run-tests-linux --python system3 automation.py run-tests-linux --python 3.5 automation.py run-tests-linux --python pypy2.7 By default, the system Python 2.7 is used. Using this functionality, I was able to identity some unexpected test failures on PyPy! Included in the feature is support for running with alternate filesystems. You can simply pass --filesystem to the command to specify the type of filesystem to run tests on. When the ephemeral instance is started, a new filesystem will be created and tests will run from it: wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4 We also support multiple Linux distributions: $ automation.py run-tests-linux --distro debian9 total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4% $ automation.py run-tests-linux --distro ubuntu18.04 total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4% $ automation.py run-tests-linux --distro ubuntu18.10 total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9% $ automation.py run-tests-linux --distro ubuntu19.04 total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0% Debian and Ubuntu are supported because those are what I use and am most familiar with. It should be easy enough to add support for other distros. Unlike the Windows AMIs, Linux EC2 instances bill per second. So the cost to instantiating an ephemeral instance isn't as severe. That being said, there is some overhead, as it takes several dozen seconds for the instance to boot, push local changes, and build Mercurial. During this time, the instance is largely CPU idle and wasting money. Even with this inefficiency, running tests is relatively cheap: $0.15-$0.25 per full test run. A machine running tests as efficiently as these EC2 instances would cost say $6,000, so you can run the test harness a >20,000 times for the cost of an equivalent machine. Running tests in EC2 is almost certainly cheaper than buying a beefy machine for developers to use :) # no-check-commit because foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6319

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Sat, 27 Apr 2019 11:48:26 -0700
parents	195dcc10b3d7
children	92a99822e731

rev	line source
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # aws.py - Automation code for Amazon Web Services
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 # no-check-code because Python 3 native.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10 import contextlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11 import copy
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12 import hashlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13 import json
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 import os
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 import pathlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	16 import subprocess
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 import time
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 import boto3
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	20 import botocore.exceptions
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	21
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	22 from .linux import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	23 BOOTSTRAP_DEBIAN,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	24 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	25 from .ssh import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	26 exec_command as ssh_exec_command,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	27 wait_for_ssh,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	28 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 from .winrm import (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 run_powershell,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 wait_for_winrm,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	34
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	35 SOURCE_ROOT = pathlib.Path(os.path.abspath(__file__)).parent.parent.parent.parent
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	37 INSTALL_WINDOWS_DEPENDENCIES = (SOURCE_ROOT / 'contrib' /
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	38 'install-windows-dependencies.ps1')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	39
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	40
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	41 INSTANCE_TYPES_WITH_STORAGE = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	42 'c5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	43 'd2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	44 'h1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	45 'i3',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	46 'm5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	47 'm5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	48 'r5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	49 'r5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	50 'x1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	51 'z1d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	52 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	53
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	54
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	55 DEBIAN_ACCOUNT_ID = '379101102735'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	56 UBUNTU_ACCOUNT_ID = '099720109477'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	57
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	58
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	59 KEY_PAIRS = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	60 'automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	61 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	62
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	63
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	64 SECURITY_GROUPS = {
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	65 'linux-dev-1': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	66 'description': 'Mercurial Linux instances that perform build/test automation',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	67 'ingress': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	68 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	69 'FromPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	70 'ToPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	71 'IpProtocol': 'tcp',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	72 'IpRanges': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	73 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	74 'CidrIp': '0.0.0.0/0',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	75 'Description': 'SSH from entire Internet',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	76 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	77 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	78 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	79 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	80 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	81 'windows-dev-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	82 'description': 'Mercurial Windows instances that perform build automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	83 'ingress': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	84 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	85 'FromPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 'ToPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 'Description': 'SSH from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 'FromPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97 'ToPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 'Description': 'RDP from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 'FromPort': 5985,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109 'ToPort': 5986,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 'Description': 'PowerShell Remoting (Windows Remote Management)',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 IAM_ROLES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 'ephemeral-ec2-role-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125 'description': 'Mercurial temporary EC2 instances',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	126 'policy_arns': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133 ASSUME_ROLE_POLICY_DOCUMENT = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 "Version": "2012-10-17",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136 "Statement": [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 "Effect": "Allow",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 "Principal": {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	140 "Service": "ec2.amazonaws.com"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	142 "Action": "sts:AssumeRole"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 '''.strip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	149 IAM_INSTANCE_PROFILES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150 'ephemeral-ec2-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151 'roles': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152 'ephemeral-ec2-role-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	153 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	154 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	155 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	156
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	157
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	158 # User Data for Windows EC2 instance. Mainly used to set the password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	159 # and configure WinRM.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160 # Inspired by the User Data script used by Packer
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	161 # (from https://www.packer.io/intro/getting-started/build-image.html).
42064 0e9066db5e44 automation: use raw strings when there are backslashes Gregory Szorc <gregory.szorc@gmail.com> parents: 42024 diff changeset	162 WINDOWS_USER_DATA = r'''
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	163 <powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	165 # TODO enable this once we figure out what is failing.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	166 #$ErrorActionPreference = "stop"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	167
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168 # Set administrator password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169 net user Administrator "%s"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170 wmic useraccount where "name='Administrator'" set PasswordExpires=FALSE
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172 # First, make sure WinRM can't be connected to
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	173 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new enable=yes action=block
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	174
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	175 # Delete any existing WinRM listeners
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176 winrm delete winrm/config/listener?Address=*+Transport=HTTP 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	177 winrm delete winrm/config/listener?Address=*+Transport=HTTPS 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	178
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	179 # Create a new WinRM listener and configure
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	180 winrm create winrm/config/listener?Address=*+Transport=HTTP
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	181 winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="0"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182 winrm set winrm/config '@{MaxTimeoutms="7200000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	183 winrm set winrm/config/service '@{AllowUnencrypted="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	184 winrm set winrm/config/service '@{MaxConcurrentOperationsPerUser="12000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	185 winrm set winrm/config/service/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	186 winrm set winrm/config/client/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	187
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	188 # Configure UAC to allow privilege elevation in remote shells
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	189 $Key = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	190 $Setting = 'LocalAccountTokenFilterPolicy'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	191 Set-ItemProperty -Path $Key -Name $Setting -Value 1 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	192
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	193 # Configure and restart the WinRM Service; Enable the required firewall exception
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	194 Stop-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	195 Set-Service -Name WinRM -StartupType Automatic
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	196 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new action=allow localip=any remoteip=any
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	197 Start-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	198
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	199 # Disable firewall on private network interfaces so prompts don't appear.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	200 Set-NetFirewallProfile -Name private -Enabled false
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201 </powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	202 '''.lstrip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	203
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	205 WINDOWS_BOOTSTRAP_POWERSHELL = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	206 Write-Output "installing PowerShell dependencies"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	207 Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	208 Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209 Install-Module -Name OpenSSHUtils -RequiredVersion 0.0.2.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	210
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 Write-Output "installing OpenSSL server"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212 Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213 # Various tools will attempt to use older versions of .NET. So we enable
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214 # the feature that provides them so it doesn't have to be auto-enabled
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	215 # later.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	216 Write-Output "enabling .NET Framework feature"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	217 Install-WindowsFeature -Name Net-Framework-Core
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	218 '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221 class AWSConnection:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222 """Manages the state of a connection with AWS."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	223
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	224 def __init__(self, automation, region: str, ensure_ec2_state: bool=True):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	225 self.automation = automation
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	226 self.local_state_path = automation.state_path
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	227
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	228 self.prefix = 'hg-'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	229
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	230 self.session = boto3.session.Session(region_name=region)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	231 self.ec2client = self.session.client('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	232 self.ec2resource = self.session.resource('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233 self.iamclient = self.session.client('iam')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 self.iamresource = self.session.resource('iam')
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	235 self.security_groups = {}
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	237 if ensure_ec2_state:
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	238 ensure_key_pairs(automation.state_path, self.ec2resource)
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	239 self.security_groups = ensure_security_groups(self.ec2resource)
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	240 ensure_iam_state(self.iamclient, self.iamresource)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	241
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	242 def key_pair_path_private(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	243 """Path to a key pair private key file."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	244 return self.local_state_path / 'keys' / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	245
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	246 def key_pair_path_public(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	247 return self.local_state_path / 'keys' / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	248
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	249
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	250 def rsa_key_fingerprint(p: pathlib.Path):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	251 """Compute the fingerprint of an RSA private key."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	252
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	253 # TODO use rsa package.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	254 res = subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	255 ['openssl', 'pkcs8', '-in', str(p), '-nocrypt', '-topk8',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	256 '-outform', 'DER'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257 capture_output=True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 check=True)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	260 sha1 = hashlib.sha1(res.stdout).hexdigest()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	261 return ':'.join(a + b for a, b in zip(sha1[::2], sha1[1::2]))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	262
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	263
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	264 def ensure_key_pairs(state_path: pathlib.Path, ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	265 remote_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	266
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	267 for kpi in ec2resource.key_pairs.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	268 if kpi.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	269 remote_existing[kpi.name[len(prefix):]] = kpi.key_fingerprint
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	271 # Validate that we have these keys locally.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 key_path = state_path / 'keys'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 key_path.mkdir(exist_ok=True, mode=0o700)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275 def remove_remote(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 print('deleting key pair %s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 key = ec2resource.KeyPair(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278 key.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 def remove_local(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284 print('removing %s' % pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286 print('removing %s' % priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 local_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 for f in sorted(os.listdir(key_path)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292 if not f.startswith('keypair-') or not f.endswith('.pub'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	293 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 name = f[len('keypair-'):-len('.pub')]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297 pub_full = key_path / f
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 with open(pub_full, 'r', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301 data = fh.read()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303 if not data.startswith('ssh-rsa '):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304 print('unexpected format for key pair file: %s; removing' %
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305 pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	309
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 local_existing[name] = rsa_key_fingerprint(priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312 for name in sorted(set(remote_existing) \| set(local_existing)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313 if name not in local_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315 print('remote key %s does not exist locally' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	316 remove_remote(actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319 elif name not in remote_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 print('local key %s does not exist remotely' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324 elif remote_existing[name] != local_existing[name]:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	325 print('key fingerprint mismatch for %s; '
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 'removing from local and remote' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328 remove_remote('%s%s' % (prefix, name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	330 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	332 missing = KEY_PAIRS - set(remote_existing)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	333
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	334 for name in sorted(missing):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	335 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	336 print('creating key pair %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	337
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	338 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	339 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	340
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	341 kp = ec2resource.create_key_pair(KeyName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 with priv_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344 fh.write(kp.key_material)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345 fh.write('\n')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	347 priv_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	349 # SSH public key can be extracted via `ssh-keygen`.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	350 with pub_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	351 subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	352 ['ssh-keygen', '-y', '-f', str(priv_full)],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	353 stdout=fh,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	354 check=True)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	355
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	356 pub_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359 def delete_instance_profile(profile):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360 for role in profile.roles:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	361 print('removing role %s from instance profile %s' % (role.name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	362 profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	363 profile.remove_role(RoleName=role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	364
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365 print('deleting instance profile %s' % profile.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	366 profile.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	367
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	368
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	369 def ensure_iam_state(iamclient, iamresource, prefix='hg-'):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370 """Ensure IAM state is in sync with our canonical definition."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372 remote_profiles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	373
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	374 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	375 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376 remote_profiles[profile.name[len(prefix):]] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378 for name in sorted(set(remote_profiles) - set(IAM_INSTANCE_PROFILES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379 delete_instance_profile(remote_profiles[name])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380 del remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382 remote_roles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385 if role.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	386 remote_roles[role.name[len(prefix):]] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	387
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	388 for name in sorted(set(remote_roles) - set(IAM_ROLES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	389 role = remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	390
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	391 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	392 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	393 del remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	394
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	395 # We've purged remote state that doesn't belong. Create missing
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396 # instance profiles and roles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397 for name in sorted(set(IAM_INSTANCE_PROFILES) - set(remote_profiles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399 print('creating IAM instance profile %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	400
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	401 profile = iamresource.create_instance_profile(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	402 InstanceProfileName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	403 remote_profiles[name] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	405 waiter = iamclient.get_waiter('instance_profile_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	406 waiter.wait(InstanceProfileName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	407 print('IAM instance profile %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	408
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	409 for name in sorted(set(IAM_ROLES) - set(remote_roles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	410 entry = IAM_ROLES[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	411
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	412 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	413 print('creating IAM role %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	414
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	415 role = iamresource.create_role(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	416 RoleName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	417 Description=entry['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	418 AssumeRolePolicyDocument=ASSUME_ROLE_POLICY_DOCUMENT,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	419 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	420
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	421 waiter = iamclient.get_waiter('role_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	422 waiter.wait(RoleName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	423 print('IAM role %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	424
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	425 remote_roles[name] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	426
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	427 for arn in entry['policy_arns']:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	428 print('attaching policy %s to %s' % (arn, role.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	429 role.attach_policy(PolicyArn=arn)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431 # Now reconcile state of profiles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 for name, meta in sorted(IAM_INSTANCE_PROFILES.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433 profile = remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434 wanted = {'%s%s' % (prefix, role) for role in meta['roles']}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	435 have = {role.name for role in profile.roles}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	436
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	437 for role in sorted(have - wanted):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438 print('removing role %s from %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439 profile.remove_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441 for role in sorted(wanted - have):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442 print('adding role %s to %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	443 profile.add_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	444
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	446 def find_image(ec2resource, owner_id, name):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	447 """Find an AMI by its owner ID and name."""
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	448
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	449 images = ec2resource.images.filter(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	450 Filters=[
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	451 {
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	452 'Name': 'owner-id',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	453 'Values': [owner_id],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	454 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	455 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	456 'Name': 'state',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	457 'Values': ['available'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	458 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	460 'Name': 'image-type',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461 'Values': ['machine'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	462 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	463 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	464 'Name': 'name',
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	465 'Values': [name],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	466 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	467 ])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	468
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	469 for image in images:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	470 return image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	471
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	472 raise Exception('unable to find image for %s' % name)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	473
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	474
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	475 def ensure_security_groups(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	476 """Ensure all necessary Mercurial security groups are present.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	477
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	478 All security groups are prefixed with ``hg-`` by default. Any security
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	479 groups having this prefix but aren't in our list are deleted.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	480 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	481 existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	482
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	483 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	484 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	485 existing[group.group_name[len(prefix):]] = group
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	486
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	487 purge = set(existing) - set(SECURITY_GROUPS)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	488
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	489 for name in sorted(purge):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	490 group = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	491 print('removing legacy security group: %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	492 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	493
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	494 security_groups = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	495
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	496 for name, group in sorted(SECURITY_GROUPS.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	497 if name in existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	498 security_groups[name] = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	499 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	500
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	501 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	502 print('adding security group %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	503
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	504 group_res = ec2resource.create_security_group(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	505 Description=group['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	506 GroupName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	507 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	508
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	509 group_res.authorize_ingress(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	510 IpPermissions=group['ingress'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	512
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	513 security_groups[name] = group_res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	514
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	515 return security_groups
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	516
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	517
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	518 def terminate_ec2_instances(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	519 """Terminate all EC2 instances managed by us."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	520 waiting = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	521
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	522 for instance in ec2resource.instances.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	523 if instance.state['Name'] == 'terminated':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	524 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	525
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	526 for tag in instance.tags or []:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	527 if tag['Key'] == 'Name' and tag['Value'].startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	528 print('terminating %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	529 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	530 waiting.append(instance)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	531
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	532 for instance in waiting:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	533 instance.wait_until_terminated()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	534
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	535
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	536 def remove_resources(c, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	537 """Purge all of our resources in this EC2 region."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	538 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	539 iamresource = c.iamresource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	540
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	541 terminate_ec2_instances(ec2resource, prefix=prefix)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542
42302 730edbd836d8 automation: only iterate over our AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 42064 diff changeset	543 for image in ec2resource.images.filter(Owners=['self']):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544 if image.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545 remove_ami(ec2resource, image)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	547 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	548 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 print('removing security group %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	552 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	553 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	554 delete_instance_profile(profile)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	555
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	556 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	557 if role.name.startswith(prefix):
42303 fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	558 for p in role.attached_policies.all():
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	559 print('detaching policy %s from %s' % (p.arn, role.name))
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	560 role.detach_policy(PolicyArn=p.arn)
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	561
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	562 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	563 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	564
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	565
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	566 def wait_for_ip_addresses(instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567 """Wait for the public IP addresses of an iterable of instances."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	568 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	570 if not instance.public_ip_address:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 instance.reload()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	574
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	575 print('public IP address for %s: %s' % (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	576 instance.id, instance.public_ip_address))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	577 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	578
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	580 def remove_ami(ec2resource, image):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	581 """Remove an AMI and its underlying snapshots."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	582 snapshots = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	583
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	584 for device in image.block_device_mappings:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	585 if 'Ebs' in device:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586 snapshots.append(ec2resource.Snapshot(device['Ebs']['SnapshotId']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	587
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	588 print('deregistering %s' % image.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	589 image.deregister()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	590
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	591 for snapshot in snapshots:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	592 print('deleting snapshot %s' % snapshot.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	593 snapshot.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	594
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	596 def wait_for_ssm(ssmclient, instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	597 """Wait for SSM to come online for an iterable of instance IDs."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	598 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	599 res = ssmclient.describe_instance_information(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	600 Filters=[
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	601 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	602 'Key': 'InstanceIds',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	603 'Values': [i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	604 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	605 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	606 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	607
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	608 available = len(res['InstanceInformationList'])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	609 wanted = len(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	610
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	611 print('%d/%d instances available in SSM' % (available, wanted))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	612
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	613 if available == wanted:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	614 return
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	615
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	616 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	617
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	618
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	619 def run_ssm_command(ssmclient, instances, document_name, parameters):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	620 """Run a PowerShell script on an EC2 instance."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	621
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	622 res = ssmclient.send_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	623 InstanceIds=[i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	624 DocumentName=document_name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	625 Parameters=parameters,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	626 CloudWatchOutputConfig={
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	627 'CloudWatchOutputEnabled': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	628 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	629 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	630
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	631 command_id = res['Command']['CommandId']
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	632
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	633 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	634 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	635 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	636 res = ssmclient.get_command_invocation(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	637 CommandId=command_id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	638 InstanceId=instance.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	639 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	640 except botocore.exceptions.ClientError as e:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	641 if e.response['Error']['Code'] == 'InvocationDoesNotExist':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	642 print('could not find SSM command invocation; waiting')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	643 time.sleep(1)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	644 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	645 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	646 raise
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	647
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	648 if res['Status'] == 'Success':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	649 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	650 elif res['Status'] in ('Pending', 'InProgress', 'Delayed'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	651 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	652 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	653 raise Exception('command failed on %s: %s' % (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	654 instance.id, res['Status']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	655
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	656
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	657 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	658 def temporary_ec2_instances(ec2resource, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	659 """Create temporary EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	660
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	661 This is a proxy to ``ec2client.run_instances(**config)`` that takes care of
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	662 managing the lifecycle of the instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	663
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	664 When the context manager exits, the instances are terminated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	665
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	666 The context manager evaluates to the list of data structures
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	667 describing each created instance. The instances may not be available
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	668 for work immediately: it is up to the caller to wait for the instance
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	669 to start responding.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	670 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	671
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	672 ids = None
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	673
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	674 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	675 res = ec2resource.create_instances(**config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	676
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	677 ids = [i.id for i in res]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	678 print('started instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	679
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	680 yield res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	681 finally:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	682 if ids:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	683 print('terminating instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	684 for instance in res:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	685 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	686 print('terminated %d instances' % len(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	687
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	688
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	689 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	690 def create_temp_windows_ec2_instances(c: AWSConnection, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	691 """Create temporary Windows EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	692
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	693 This is a higher-level wrapper around ``create_temp_ec2_instances()`` that
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	694 configures the Windows instance for Windows Remote Management. The emitted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	695 instances will have a ``winrm_client`` attribute containing a
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	696 ``pypsrp.client.Client`` instance bound to the instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	697 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	698 if 'IamInstanceProfile' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	699 raise ValueError('IamInstanceProfile cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	700 if 'UserData' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	701 raise ValueError('UserData cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	702
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	703 password = c.automation.default_password()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	704
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	705 config = copy.deepcopy(config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	706 config['IamInstanceProfile'] = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	707 'Name': 'hg-ephemeral-ec2-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	708 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	709 config.setdefault('TagSpecifications', []).append({
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	710 'ResourceType': 'instance',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	711 'Tags': [{'Key': 'Name', 'Value': 'hg-temp-windows'}],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	712 })
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	713 config['UserData'] = WINDOWS_USER_DATA % password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	714
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	715 with temporary_ec2_instances(c.ec2resource, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	716 wait_for_ip_addresses(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	717
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	718 print('waiting for Windows Remote Management service...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	719
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	720 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	721 client = wait_for_winrm(instance.public_ip_address, 'Administrator', password)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	722 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	724
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	725 yield instances
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	726
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	727
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	728 def resolve_fingerprint(fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	729 fingerprint = json.dumps(fingerprint, sort_keys=True)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	730 return hashlib.sha256(fingerprint.encode('utf-8')).hexdigest()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	731
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	732
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	733 def find_and_reconcile_image(ec2resource, name, fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	734 """Attempt to find an existing EC2 AMI with a name and fingerprint.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	735
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	736 If an image with the specified fingerprint is found, it is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	737 Otherwise None is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	738
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	739 Existing images for the specified name that don't have the specified
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	740 fingerprint or are missing required metadata or deleted.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	741 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	742 # Find existing AMIs with this name and delete the ones that are invalid.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	743 # Store a reference to a good image so it can be returned one the
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	744 # image state is reconciled.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	745 images = ec2resource.images.filter(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	746 Filters=[{'Name': 'name', 'Values': [name]}])
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	747
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	748 existing_image = None
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	749
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	750 for image in images:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	751 if image.tags is None:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	752 print('image %s for %s lacks required tags; removing' % (
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	753 image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	754 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	755 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	756 tags = {t['Key']: t['Value'] for t in image.tags}
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	757
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	758 if tags.get('HGIMAGEFINGERPRINT') == fingerprint:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	759 existing_image = image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	760 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	761 print('image %s for %s has wrong fingerprint; removing' % (
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	762 image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	763 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	764
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	765 return existing_image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	766
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	767
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	768 def create_ami_from_instance(ec2client, instance, name, description,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	769 fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	770 """Create an AMI from a running instance.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	771
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	772 Returns the ``ec2resource.Image`` representing the created AMI.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	773 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	774 instance.stop()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	775
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	776 ec2client.get_waiter('instance_stopped').wait(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	777 InstanceIds=[instance.id],
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	778 WaiterConfig={
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	779 'Delay': 5,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	780 })
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	781 print('%s is stopped' % instance.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	782
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	783 image = instance.create_image(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	784 Name=name,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	785 Description=description,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	786 )
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	787
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	788 image.create_tags(Tags=[
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	789 {
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	790 'Key': 'HGIMAGEFINGERPRINT',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	791 'Value': fingerprint,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	792 },
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	793 ])
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	794
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	795 print('waiting for image %s' % image.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	796
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	797 ec2client.get_waiter('image_available').wait(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	798 ImageIds=[image.id],
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	799 )
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	800
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	801 print('image %s available as %s' % (image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	802
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	803 return image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	804
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	805
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	806 def ensure_linux_dev_ami(c: AWSConnection, distro='debian9', prefix='hg-'):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	807 """Ensures a Linux development AMI is available and up-to-date.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	808
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	809 Returns an ``ec2.Image`` of either an existing AMI or a newly-built one.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	810 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	811 ec2client = c.ec2client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	812 ec2resource = c.ec2resource
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	813
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	814 name = '%s%s-%s' % (prefix, 'linux-dev', distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	815
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	816 if distro == 'debian9':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	817 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	818 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	819 DEBIAN_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	820 'debian-stretch-hvm-x86_64-gp2-2019-02-19-26620',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	821 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	822 ssh_username = 'admin'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	823 elif distro == 'ubuntu18.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	824 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	825 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	826 UBUNTU_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	827 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190403',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	828 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	829 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	830 elif distro == 'ubuntu18.10':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	831 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	832 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	833 UBUNTU_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	834 'ubuntu/images/hvm-ssd/ubuntu-cosmic-18.10-amd64-server-20190402',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	835 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	836 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	837 elif distro == 'ubuntu19.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	838 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	839 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	840 UBUNTU_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	841 'ubuntu/images/hvm-ssd/ubuntu-disco-19.04-amd64-server-20190417',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	842 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	843 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	844 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	845 raise ValueError('unsupported Linux distro: %s' % distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	846
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	847 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	848 'BlockDeviceMappings': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	849 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	850 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	851 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	852 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	853 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	854 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	855 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	856 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	857 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	858 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	859 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	860 'InstanceInitiatedShutdownBehavior': 'stop',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	861 # 8 VCPUs for compiling Python.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	862 'InstanceType': 't3.2xlarge',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	863 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	864 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	865 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	866 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	867 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	868
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	869 requirements2_path = (pathlib.Path(__file__).parent.parent /
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	870 'linux-requirements-py2.txt')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	871 requirements3_path = (pathlib.Path(__file__).parent.parent /
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	872 'linux-requirements-py3.txt')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	873 with requirements2_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	874 requirements2 = fh.read()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	875 with requirements3_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	876 requirements3 = fh.read()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	877
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	878 # Compute a deterministic fingerprint to determine whether image needs to
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	879 # be regenerated.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	880 fingerprint = resolve_fingerprint({
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	881 'instance_config': config,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	882 'bootstrap_script': BOOTSTRAP_DEBIAN,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	883 'requirements_py2': requirements2,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	884 'requirements_py3': requirements3,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	885 })
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	886
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	887 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	888
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	889 if existing_image:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	890 return existing_image
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	891
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	892 print('no suitable %s image found; creating one...' % name)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	893
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	894 with temporary_ec2_instances(ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	895 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	896
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	897 instance = instances[0]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	898
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	899 client = wait_for_ssh(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	900 instance.public_ip_address, 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	901 username=ssh_username,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	902 key_filename=str(c.key_pair_path_private('automation')))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	903
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	904 home = '/home/%s' % ssh_username
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	905
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	906 with client:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	907 print('connecting to SSH server')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	908 sftp = client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	909
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	910 print('uploading bootstrap files')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	911 with sftp.open('%s/bootstrap' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	912 fh.write(BOOTSTRAP_DEBIAN)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	913 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	914
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	915 with sftp.open('%s/requirements-py2.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	916 fh.write(requirements2)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	917 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	918
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	919 with sftp.open('%s/requirements-py3.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	920 fh.write(requirements3)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	921 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	922
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	923 print('executing bootstrap')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	924 chan, stdin, stdout = ssh_exec_command(client,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	925 '%s/bootstrap' % home)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	926 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	927
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	928 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	929 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	930
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	931 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	932 if res:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	933 raise Exception('non-0 exit from bootstrap: %d' % res)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	934
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	935 print('bootstrap completed; stopping %s to create %s' % (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	936 instance.id, name))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	937
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	938 return create_ami_from_instance(ec2client, instance, name,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	939 'Mercurial Linux development environment',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	940 fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	941
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	942
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	943 @contextlib.contextmanager
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	944 def temporary_linux_dev_instances(c: AWSConnection, image, instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	945 prefix='hg-', ensure_extra_volume=False):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	946 """Create temporary Linux development EC2 instances.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	947
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	948 Context manager resolves to a list of ``ec2.Instance`` that were created
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	949 and are running.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	950
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	951 ``ensure_extra_volume`` can be set to ``True`` to require that instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	952 have a 2nd storage volume available other than the primary AMI volume.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	953 For instance types with instance storage, this does nothing special.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	954 But for instance types without instance storage, an additional EBS volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	955 will be added to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	956
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	957 Instances have an ``ssh_client`` attribute containing a paramiko SSHClient
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	958 instance bound to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	959
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	960 Instances have an ``ssh_private_key_path`` attributing containing the
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	961 str path to the SSH private key to connect to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	962 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	963
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	964 block_device_mappings = [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	965 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	966 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	967 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	968 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	969 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	970 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	971 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	972 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	973 ]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	974
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	975 # This is not an exhaustive list of instance types having instance storage.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	976 # But
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	977 if (ensure_extra_volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	978 and not instance_type.startswith(tuple(INSTANCE_TYPES_WITH_STORAGE))):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	979 main_device = block_device_mappings[0]['DeviceName']
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	980
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	981 if main_device == 'xvda':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	982 second_device = 'xvdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	983 elif main_device == '/dev/sda1':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	984 second_device = '/dev/sdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	985 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	986 raise ValueError('unhandled primary EBS device name: %s' %
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	987 main_device)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	988
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	989 block_device_mappings.append({
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	990 'DeviceName': second_device,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	991 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	992 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	993 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	994 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	995 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	996 })
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	997
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	998 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	999 'BlockDeviceMappings': block_device_mappings,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1000 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1001 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1002 'InstanceInitiatedShutdownBehavior': 'terminate',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1003 'InstanceType': instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1004 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1005 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1006 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1007 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1008 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1009
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1010 with temporary_ec2_instances(c.ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1011 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1012
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1013 ssh_private_key_path = str(c.key_pair_path_private('automation'))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1014
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1015 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1016 client = wait_for_ssh(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1017 instance.public_ip_address, 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1018 username='hg',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1019 key_filename=ssh_private_key_path)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1020
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1021 instance.ssh_client = client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1022 instance.ssh_private_key_path = ssh_private_key_path
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1023
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1024 try:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1025 yield instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1026 finally:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1027 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1028 instance.ssh_client.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1029
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1030
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1031 def ensure_windows_dev_ami(c: AWSConnection, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1032 """Ensure Windows Development AMI is available and up-to-date.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1033
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1034 If necessary, a modern AMI will be built by starting a temporary EC2
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1035 instance and bootstrapping it.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1036
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1037 Obsolete AMIs will be deleted so there is only a single AMI having the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1038 desired name.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1039
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1040 Returns an ``ec2.Image`` of either an existing AMI or a newly-built
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1041 one.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1042 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1043 ec2client = c.ec2client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1044 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1045 ssmclient = c.session.client('ssm')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1046
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1047 name = '%s%s' % (prefix, 'windows-dev')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1048
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1049 image = find_image(ec2resource,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1050 '801119661308',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1051 'Windows_Server-2019-English-Full-Base-2019.02.13')
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1052
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1053 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1054 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1055 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1056 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1057 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1058 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1059 'VolumeSize': 32,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1060 'VolumeType': 'gp2',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1061 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1062 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1063 ],
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1064 'ImageId': image.id,
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1065 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1066 'InstanceType': 't3.medium',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1067 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1068 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1069 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1070 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1071 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1072
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1073 commands = [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1074 # Need to start the service so sshd_config is generated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1075 'Start-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1076 'Write-Output "modifying sshd_config"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1077 r'$content = Get-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1078 '$content = $content -replace "Match Group administrators","" -replace "AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys",""',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1079 r'$content \| Set-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1080 'Import-Module OpenSSHUtils',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1081 r'Repair-SshdConfigPermission C:\ProgramData\ssh\sshd_config -Confirm:$false',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1082 'Restart-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1083 'Write-Output "installing OpenSSL client"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1084 'Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1085 'Set-Service -Name sshd -StartupType "Automatic"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1086 'Write-Output "OpenSSH server running"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1087 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1088
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1089 with INSTALL_WINDOWS_DEPENDENCIES.open('r', encoding='utf-8') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1090 commands.extend(l.rstrip() for l in fh)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1091
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1092 # Disable Windows Defender when bootstrapping because it just slows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1093 # things down.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1094 commands.insert(0, 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1095 commands.append('Set-MpPreference -DisableRealtimeMonitoring $false')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1096
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1097 # Compute a deterministic fingerprint to determine whether image needs
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1098 # to be regenerated.
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1099 fingerprint = resolve_fingerprint({
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1100 'instance_config': config,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1101 'user_data': WINDOWS_USER_DATA,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1102 'initial_bootstrap': WINDOWS_BOOTSTRAP_POWERSHELL,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1103 'bootstrap_commands': commands,
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1104 })
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1105
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1106 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1107
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1108 if existing_image:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1109 return existing_image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1110
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1111 print('no suitable Windows development image found; creating one...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1112
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1113 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1114 assert len(instances) == 1
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1115 instance = instances[0]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1116
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1117 wait_for_ssm(ssmclient, [instance])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1118
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1119 # On first boot, install various Windows updates.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1120 # We would ideally use PowerShell Remoting for this. However, there are
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1121 # trust issues that make it difficult to invoke Windows Update
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1122 # remotely. So we use SSM, which has a mechanism for running Windows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1123 # Update.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1124 print('installing Windows features...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1125 run_ssm_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1126 ssmclient,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1127 [instance],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1128 'AWS-RunPowerShellScript',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1129 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1130 'commands': WINDOWS_BOOTSTRAP_POWERSHELL.split('\n'),
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1131 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1132 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1133
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1134 # Reboot so all updates are fully applied.
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1135 #
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1136 # We don't use instance.reboot() here because it is asynchronous and
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1137 # we don't know when exactly the instance has rebooted. It could take
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1138 # a while to stop and we may start trying to interact with the instance
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1139 # before it has rebooted.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1140 print('rebooting instance %s' % instance.id)
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1141 instance.stop()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1142 ec2client.get_waiter('instance_stopped').wait(
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1143 InstanceIds=[instance.id],
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1144 WaiterConfig={
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1145 'Delay': 5,
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1146 })
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1147
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1148 instance.start()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1149 wait_for_ip_addresses([instance])
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1150
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1151 # There is a race condition here between the User Data PS script running
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1152 # and us connecting to WinRM. This can manifest as
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1153 # "AuthorizationManager check failed" failures during run_powershell().
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1154 # TODO figure out a workaround.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1155
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1156 print('waiting for Windows Remote Management to come back...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1157 client = wait_for_winrm(instance.public_ip_address, 'Administrator',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1158 c.automation.default_password())
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1159 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1160 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1161
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1162 print('bootstrapping instance...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1163 run_powershell(instance.winrm_client, '\n'.join(commands))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1164
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1165 print('bootstrap completed; stopping %s to create image' % instance.id)
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1166 return create_ami_from_instance(ec2client, instance, name,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1167 'Mercurial Windows development environment',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1168 fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1169
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1170
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1171 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1172 def temporary_windows_dev_instances(c: AWSConnection, image, instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1173 prefix='hg-', disable_antivirus=False):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1174 """Create a temporary Windows development EC2 instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1175
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1176 Context manager resolves to the list of ``EC2.Instance`` that were created.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1177 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1178 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1179 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1180 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1181 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1182 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1183 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1184 'VolumeSize': 32,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1185 'VolumeType': 'gp2',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1186 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1187 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1188 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1189 'ImageId': image.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1190 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1191 'InstanceType': instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1192 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1193 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1194 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1195 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1196 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1197
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1198 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1199 if disable_antivirus:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1200 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1201 run_powershell(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1202 instance.winrm_client,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1203 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1204
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1205 yield instances

Mercurial > public > mercurial-scm > hg-stable

annotate contrib/automation/hgautomation/aws.py @ 42312:65b3ef162b39