mercurial-scm/hg-stable: contrib/automation/hgautomation/aws.py annotate

annotate contrib/automation/hgautomation/aws.py @ 43020:d1d919f679f7

automation: support and use Debian Buster by default While Debian Buster (Debian 10) was released in July, the AWS AMIs were not published until mid September. This commit teaches the automation system to create AMIs for Debian Buster. Since Debian Buster is the new stable Debian release, we make it the default distribution for automation. Differential Revision: https://phab.mercurial-scm.org/D6917

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Sun, 29 Sep 2019 11:29:25 -0700
parents	6952d42f9158
children	2372284d9457

rev	line source
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # aws.py - Automation code for Amazon Web Services
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 # no-check-code because Python 3 native.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10 import contextlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11 import copy
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12 import hashlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13 import json
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 import os
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 import pathlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	16 import subprocess
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 import time
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 import boto3
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	20 import botocore.exceptions
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	21
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	22 from .linux import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	23 BOOTSTRAP_DEBIAN,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	24 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	25 from .ssh import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	26 exec_command as ssh_exec_command,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	27 wait_for_ssh,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	28 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 from .winrm import (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 run_powershell,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 wait_for_winrm,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	34
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	35 SOURCE_ROOT = pathlib.Path(os.path.abspath(__file__)).parent.parent.parent.parent
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	37 INSTALL_WINDOWS_DEPENDENCIES = (SOURCE_ROOT / 'contrib' /
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	38 'install-windows-dependencies.ps1')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	39
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	40
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	41 INSTANCE_TYPES_WITH_STORAGE = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	42 'c5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	43 'd2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	44 'h1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	45 'i3',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	46 'm5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	47 'm5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	48 'r5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	49 'r5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	50 'x1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	51 'z1d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	52 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	53
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	54
42647 8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	55 AMAZON_ACCOUNT_ID = '801119661308'
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	56 DEBIAN_ACCOUNT_ID = '379101102735'
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	57 DEBIAN_ACCOUNT_ID_2 = '136693071363'
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	58 UBUNTU_ACCOUNT_ID = '099720109477'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	59
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	60
42647 8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	61 WINDOWS_BASE_IMAGE_NAME = 'Windows_Server-2019-English-Full-Base-2019.07.12'
8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	62
8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	63
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	64 KEY_PAIRS = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	65 'automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	66 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	69 SECURITY_GROUPS = {
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	70 'linux-dev-1': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	71 'description': 'Mercurial Linux instances that perform build/test automation',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	72 'ingress': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	73 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	74 'FromPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	75 'ToPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	76 'IpProtocol': 'tcp',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	77 'IpRanges': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	78 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	79 'CidrIp': '0.0.0.0/0',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	80 'Description': 'SSH from entire Internet',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	81 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	82 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	83 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	84 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	85 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 'windows-dev-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 'description': 'Mercurial Windows instances that perform build automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 'ingress': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 'FromPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 'ToPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 'Description': 'SSH from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 'FromPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 'ToPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107 'Description': 'RDP from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 'FromPort': 5985,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 'ToPort': 5986,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119 'Description': 'PowerShell Remoting (Windows Remote Management)',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	126
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 IAM_ROLES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 'ephemeral-ec2-role-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130 'description': 'Mercurial temporary EC2 instances',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131 'policy_arns': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 ASSUME_ROLE_POLICY_DOCUMENT = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	140 "Version": "2012-10-17",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141 "Statement": [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	142 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 "Effect": "Allow",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 "Principal": {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145 "Service": "ec2.amazonaws.com"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147 "Action": "sts:AssumeRole"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	149 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151 '''.strip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	153
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	154 IAM_INSTANCE_PROFILES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	155 'ephemeral-ec2-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	156 'roles': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	157 'ephemeral-ec2-role-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	158 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	159 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	161
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	162
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	163 # User Data for Windows EC2 instance. Mainly used to set the password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164 # and configure WinRM.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	165 # Inspired by the User Data script used by Packer
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	166 # (from https://www.packer.io/intro/getting-started/build-image.html).
42064 0e9066db5e44 automation: use raw strings when there are backslashes Gregory Szorc <gregory.szorc@gmail.com> parents: 42024 diff changeset	167 WINDOWS_USER_DATA = r'''
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168 <powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170 # TODO enable this once we figure out what is failing.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171 #$ErrorActionPreference = "stop"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	173 # Set administrator password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	174 net user Administrator "%s"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	175 wmic useraccount where "name='Administrator'" set PasswordExpires=FALSE
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	177 # First, make sure WinRM can't be connected to
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	178 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new enable=yes action=block
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	179
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	180 # Delete any existing WinRM listeners
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	181 winrm delete winrm/config/listener?Address=*+Transport=HTTP 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182 winrm delete winrm/config/listener?Address=*+Transport=HTTPS 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	183
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	184 # Create a new WinRM listener and configure
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	185 winrm create winrm/config/listener?Address=*+Transport=HTTP
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	186 winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="0"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	187 winrm set winrm/config '@{MaxTimeoutms="7200000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	188 winrm set winrm/config/service '@{AllowUnencrypted="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	189 winrm set winrm/config/service '@{MaxConcurrentOperationsPerUser="12000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	190 winrm set winrm/config/service/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	191 winrm set winrm/config/client/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	192
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	193 # Configure UAC to allow privilege elevation in remote shells
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	194 $Key = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	195 $Setting = 'LocalAccountTokenFilterPolicy'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	196 Set-ItemProperty -Path $Key -Name $Setting -Value 1 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	197
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	198 # Configure and restart the WinRM Service; Enable the required firewall exception
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	199 Stop-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	200 Set-Service -Name WinRM -StartupType Automatic
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new action=allow localip=any remoteip=any
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	202 Start-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	203
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204 # Disable firewall on private network interfaces so prompts don't appear.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	205 Set-NetFirewallProfile -Name private -Enabled false
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	206 </powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	207 '''.lstrip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	208
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	210 WINDOWS_BOOTSTRAP_POWERSHELL = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 Write-Output "installing PowerShell dependencies"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212 Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213 Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214 Install-Module -Name OpenSSHUtils -RequiredVersion 0.0.2.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	215
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	216 Write-Output "installing OpenSSL server"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	217 Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	218 # Various tools will attempt to use older versions of .NET. So we enable
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219 # the feature that provides them so it doesn't have to be auto-enabled
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220 # later.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221 Write-Output "enabling .NET Framework feature"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222 Install-WindowsFeature -Name Net-Framework-Core
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	223 '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	224
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	225
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	226 class AWSConnection:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	227 """Manages the state of a connection with AWS."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	228
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	229 def __init__(self, automation, region: str, ensure_ec2_state: bool=True):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	230 self.automation = automation
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	231 self.local_state_path = automation.state_path
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	232
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233 self.prefix = 'hg-'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235 self.session = boto3.session.Session(region_name=region)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236 self.ec2client = self.session.client('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	237 self.ec2resource = self.session.resource('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	238 self.iamclient = self.session.client('iam')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	239 self.iamresource = self.session.resource('iam')
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	240 self.security_groups = {}
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	241
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	242 if ensure_ec2_state:
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	243 ensure_key_pairs(automation.state_path, self.ec2resource)
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	244 self.security_groups = ensure_security_groups(self.ec2resource)
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	245 ensure_iam_state(self.iamclient, self.iamresource)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	246
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	247 def key_pair_path_private(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	248 """Path to a key pair private key file."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	249 return self.local_state_path / 'keys' / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	250
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	251 def key_pair_path_public(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	252 return self.local_state_path / 'keys' / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	253
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	254
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	255 def rsa_key_fingerprint(p: pathlib.Path):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	256 """Compute the fingerprint of an RSA private key."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 # TODO use rsa package.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259 res = subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	260 ['openssl', 'pkcs8', '-in', str(p), '-nocrypt', '-topk8',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	261 '-outform', 'DER'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	262 capture_output=True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	263 check=True)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	264
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	265 sha1 = hashlib.sha1(res.stdout).hexdigest()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	266 return ':'.join(a + b for a, b in zip(sha1[::2], sha1[1::2]))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	267
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	268
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	269 def ensure_key_pairs(state_path: pathlib.Path, ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270 remote_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	271
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 for kpi in ec2resource.key_pairs.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 if kpi.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274 remote_existing[kpi.name[len(prefix):]] = kpi.key_fingerprint
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 # Validate that we have these keys locally.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 key_path = state_path / 'keys'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278 key_path.mkdir(exist_ok=True, mode=0o700)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 def remove_remote(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 print('deleting key pair %s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 key = ec2resource.KeyPair(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283 key.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285 def remove_local(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 print('removing %s' % pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 print('removing %s' % priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	293
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294 local_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 for f in sorted(os.listdir(key_path)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297 if not f.startswith('keypair-') or not f.endswith('.pub'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 name = f[len('keypair-'):-len('.pub')]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 pub_full = key_path / f
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305 with open(pub_full, 'r', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 data = fh.read()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 if not data.startswith('ssh-rsa '):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	309 print('unexpected format for key pair file: %s; removing' %
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315 local_existing[name] = rsa_key_fingerprint(priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	316
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 for name in sorted(set(remote_existing) \| set(local_existing)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318 if name not in local_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 print('remote key %s does not exist locally' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 remove_remote(actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324 elif name not in remote_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	325 print('local key %s does not exist remotely' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329 elif remote_existing[name] != local_existing[name]:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	330 print('key fingerprint mismatch for %s; '
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331 'removing from local and remote' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	332 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	333 remove_remote('%s%s' % (prefix, name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	334 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	335 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	336
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	337 missing = KEY_PAIRS - set(remote_existing)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	338
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	339 for name in sorted(missing):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	340 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	341 print('creating key pair %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346 kp = ec2resource.create_key_pair(KeyName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	347
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348 with priv_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	349 fh.write(kp.key_material)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	350 fh.write('\n')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	351
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	352 priv_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	353
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	354 # SSH public key can be extracted via `ssh-keygen`.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	355 with pub_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	356 subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357 ['ssh-keygen', '-y', '-f', str(priv_full)],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358 stdout=fh,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359 check=True)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	361 pub_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	362
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	363
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	364 def delete_instance_profile(profile):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365 for role in profile.roles:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	366 print('removing role %s from instance profile %s' % (role.name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	367 profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	368 profile.remove_role(RoleName=role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	369
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370 print('deleting instance profile %s' % profile.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371 profile.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	373
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	374 def ensure_iam_state(iamclient, iamresource, prefix='hg-'):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	375 """Ensure IAM state is in sync with our canonical definition."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377 remote_profiles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381 remote_profiles[profile.name[len(prefix):]] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383 for name in sorted(set(remote_profiles) - set(IAM_INSTANCE_PROFILES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384 delete_instance_profile(remote_profiles[name])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385 del remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	386
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	387 remote_roles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	388
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	389 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	390 if role.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	391 remote_roles[role.name[len(prefix):]] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	392
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	393 for name in sorted(set(remote_roles) - set(IAM_ROLES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	394 role = remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	395
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398 del remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	400 # We've purged remote state that doesn't belong. Create missing
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	401 # instance profiles and roles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	402 for name in sorted(set(IAM_INSTANCE_PROFILES) - set(remote_profiles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	403 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404 print('creating IAM instance profile %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	405
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	406 profile = iamresource.create_instance_profile(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	407 InstanceProfileName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	408 remote_profiles[name] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	409
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	410 waiter = iamclient.get_waiter('instance_profile_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	411 waiter.wait(InstanceProfileName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	412 print('IAM instance profile %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	413
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	414 for name in sorted(set(IAM_ROLES) - set(remote_roles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	415 entry = IAM_ROLES[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	416
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	417 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	418 print('creating IAM role %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	419
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	420 role = iamresource.create_role(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	421 RoleName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	422 Description=entry['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	423 AssumeRolePolicyDocument=ASSUME_ROLE_POLICY_DOCUMENT,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	424 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	425
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	426 waiter = iamclient.get_waiter('role_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	427 waiter.wait(RoleName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	428 print('IAM role %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	429
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430 remote_roles[name] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 for arn in entry['policy_arns']:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433 print('attaching policy %s to %s' % (arn, role.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434 role.attach_policy(PolicyArn=arn)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	435
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	436 # Now reconcile state of profiles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	437 for name, meta in sorted(IAM_INSTANCE_PROFILES.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438 profile = remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439 wanted = {'%s%s' % (prefix, role) for role in meta['roles']}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440 have = {role.name for role in profile.roles}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442 for role in sorted(have - wanted):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	443 print('removing role %s from %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	444 profile.remove_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	446 for role in sorted(wanted - have):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	447 print('adding role %s to %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	448 profile.add_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	449
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	450
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	451 def find_image(ec2resource, owner_id, name):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	452 """Find an AMI by its owner ID and name."""
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	453
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	454 images = ec2resource.images.filter(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	455 Filters=[
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	456 {
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	457 'Name': 'owner-id',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	458 'Values': [owner_id],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	460 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461 'Name': 'state',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	462 'Values': ['available'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	463 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	464 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	465 'Name': 'image-type',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	466 'Values': ['machine'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	467 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	468 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	469 'Name': 'name',
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	470 'Values': [name],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	471 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	472 ])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	473
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	474 for image in images:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	475 return image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	476
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	477 raise Exception('unable to find image for %s' % name)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	478
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	479
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	480 def ensure_security_groups(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	481 """Ensure all necessary Mercurial security groups are present.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	482
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	483 All security groups are prefixed with ``hg-`` by default. Any security
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	484 groups having this prefix but aren't in our list are deleted.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	485 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	486 existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	487
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	488 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	489 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	490 existing[group.group_name[len(prefix):]] = group
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	491
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	492 purge = set(existing) - set(SECURITY_GROUPS)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	493
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	494 for name in sorted(purge):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	495 group = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	496 print('removing legacy security group: %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	497 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	498
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	499 security_groups = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	500
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	501 for name, group in sorted(SECURITY_GROUPS.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	502 if name in existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	503 security_groups[name] = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	504 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	505
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	506 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	507 print('adding security group %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	508
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	509 group_res = ec2resource.create_security_group(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	510 Description=group['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511 GroupName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	512 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	513
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	514 group_res.authorize_ingress(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	515 IpPermissions=group['ingress'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	516 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	517
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	518 security_groups[name] = group_res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	519
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	520 return security_groups
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	521
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	522
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	523 def terminate_ec2_instances(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	524 """Terminate all EC2 instances managed by us."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	525 waiting = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	526
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	527 for instance in ec2resource.instances.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	528 if instance.state['Name'] == 'terminated':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	529 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	530
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	531 for tag in instance.tags or []:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	532 if tag['Key'] == 'Name' and tag['Value'].startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	533 print('terminating %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	534 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	535 waiting.append(instance)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	536
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	537 for instance in waiting:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	538 instance.wait_until_terminated()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	539
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	540
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	541 def remove_resources(c, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542 """Purge all of our resources in this EC2 region."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544 iamresource = c.iamresource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546 terminate_ec2_instances(ec2resource, prefix=prefix)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	547
42302 730edbd836d8 automation: only iterate over our AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 42064 diff changeset	548 for image in ec2resource.images.filter(Owners=['self']):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 if image.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550 remove_ami(ec2resource, image)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	552 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	553 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	554 print('removing security group %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	555 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	556
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	557 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	558 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	559 delete_instance_profile(profile)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	560
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	561 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	562 if role.name.startswith(prefix):
42303 fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	563 for p in role.attached_policies.all():
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	564 print('detaching policy %s from %s' % (p.arn, role.name))
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	565 role.detach_policy(PolicyArn=p.arn)
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	566
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	568 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	570
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571 def wait_for_ip_addresses(instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 """Wait for the public IP addresses of an iterable of instances."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	574 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	575 if not instance.public_ip_address:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	576 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	577 instance.reload()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	578 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	580 print('public IP address for %s: %s' % (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	581 instance.id, instance.public_ip_address))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	582 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	583
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	584
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	585 def remove_ami(ec2resource, image):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586 """Remove an AMI and its underlying snapshots."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	587 snapshots = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	588
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	589 for device in image.block_device_mappings:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	590 if 'Ebs' in device:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	591 snapshots.append(ec2resource.Snapshot(device['Ebs']['SnapshotId']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	592
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	593 print('deregistering %s' % image.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	594 image.deregister()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	596 for snapshot in snapshots:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	597 print('deleting snapshot %s' % snapshot.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	598 snapshot.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	599
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	600
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	601 def wait_for_ssm(ssmclient, instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	602 """Wait for SSM to come online for an iterable of instance IDs."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	603 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	604 res = ssmclient.describe_instance_information(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	605 Filters=[
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	606 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	607 'Key': 'InstanceIds',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	608 'Values': [i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	609 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	610 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	611 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	612
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	613 available = len(res['InstanceInformationList'])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	614 wanted = len(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	615
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	616 print('%d/%d instances available in SSM' % (available, wanted))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	617
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	618 if available == wanted:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	619 return
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	620
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	621 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	622
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	623
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	624 def run_ssm_command(ssmclient, instances, document_name, parameters):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	625 """Run a PowerShell script on an EC2 instance."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	626
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	627 res = ssmclient.send_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	628 InstanceIds=[i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	629 DocumentName=document_name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	630 Parameters=parameters,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	631 CloudWatchOutputConfig={
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	632 'CloudWatchOutputEnabled': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	633 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	634 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	635
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	636 command_id = res['Command']['CommandId']
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	637
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	638 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	639 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	640 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	641 res = ssmclient.get_command_invocation(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	642 CommandId=command_id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	643 InstanceId=instance.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	644 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	645 except botocore.exceptions.ClientError as e:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	646 if e.response['Error']['Code'] == 'InvocationDoesNotExist':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	647 print('could not find SSM command invocation; waiting')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	648 time.sleep(1)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	649 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	650 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	651 raise
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	652
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	653 if res['Status'] == 'Success':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	654 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	655 elif res['Status'] in ('Pending', 'InProgress', 'Delayed'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	656 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	657 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	658 raise Exception('command failed on %s: %s' % (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	659 instance.id, res['Status']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	660
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	661
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	662 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	663 def temporary_ec2_instances(ec2resource, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	664 """Create temporary EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	665
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	666 This is a proxy to ``ec2client.run_instances(**config)`` that takes care of
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	667 managing the lifecycle of the instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	668
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	669 When the context manager exits, the instances are terminated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	670
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	671 The context manager evaluates to the list of data structures
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	672 describing each created instance. The instances may not be available
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	673 for work immediately: it is up to the caller to wait for the instance
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	674 to start responding.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	675 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	676
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	677 ids = None
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	678
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	679 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	680 res = ec2resource.create_instances(**config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	681
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	682 ids = [i.id for i in res]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	683 print('started instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	684
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	685 yield res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	686 finally:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	687 if ids:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	688 print('terminating instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	689 for instance in res:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	690 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	691 print('terminated %d instances' % len(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	692
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	693
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	694 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	695 def create_temp_windows_ec2_instances(c: AWSConnection, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	696 """Create temporary Windows EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	697
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	698 This is a higher-level wrapper around ``create_temp_ec2_instances()`` that
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	699 configures the Windows instance for Windows Remote Management. The emitted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	700 instances will have a ``winrm_client`` attribute containing a
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	701 ``pypsrp.client.Client`` instance bound to the instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	702 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	703 if 'IamInstanceProfile' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	704 raise ValueError('IamInstanceProfile cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	705 if 'UserData' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	706 raise ValueError('UserData cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	707
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	708 password = c.automation.default_password()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	709
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	710 config = copy.deepcopy(config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	711 config['IamInstanceProfile'] = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	712 'Name': 'hg-ephemeral-ec2-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	713 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	714 config.setdefault('TagSpecifications', []).append({
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	715 'ResourceType': 'instance',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	716 'Tags': [{'Key': 'Name', 'Value': 'hg-temp-windows'}],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	717 })
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	718 config['UserData'] = WINDOWS_USER_DATA % password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	719
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	720 with temporary_ec2_instances(c.ec2resource, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	721 wait_for_ip_addresses(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	722
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723 print('waiting for Windows Remote Management service...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	724
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	725 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	726 client = wait_for_winrm(instance.public_ip_address, 'Administrator', password)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	727 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	728 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	729
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	730 yield instances
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	731
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	732
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	733 def resolve_fingerprint(fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	734 fingerprint = json.dumps(fingerprint, sort_keys=True)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	735 return hashlib.sha256(fingerprint.encode('utf-8')).hexdigest()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	736
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	737
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	738 def find_and_reconcile_image(ec2resource, name, fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	739 """Attempt to find an existing EC2 AMI with a name and fingerprint.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	740
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	741 If an image with the specified fingerprint is found, it is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	742 Otherwise None is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	743
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	744 Existing images for the specified name that don't have the specified
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	745 fingerprint or are missing required metadata or deleted.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	746 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	747 # Find existing AMIs with this name and delete the ones that are invalid.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	748 # Store a reference to a good image so it can be returned one the
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	749 # image state is reconciled.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	750 images = ec2resource.images.filter(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	751 Filters=[{'Name': 'name', 'Values': [name]}])
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	752
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	753 existing_image = None
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	754
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	755 for image in images:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	756 if image.tags is None:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	757 print('image %s for %s lacks required tags; removing' % (
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	758 image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	759 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	760 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	761 tags = {t['Key']: t['Value'] for t in image.tags}
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	762
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	763 if tags.get('HGIMAGEFINGERPRINT') == fingerprint:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	764 existing_image = image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	765 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	766 print('image %s for %s has wrong fingerprint; removing' % (
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	767 image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	768 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	769
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	770 return existing_image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	771
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	772
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	773 def create_ami_from_instance(ec2client, instance, name, description,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	774 fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	775 """Create an AMI from a running instance.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	776
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	777 Returns the ``ec2resource.Image`` representing the created AMI.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	778 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	779 instance.stop()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	780
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	781 ec2client.get_waiter('instance_stopped').wait(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	782 InstanceIds=[instance.id],
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	783 WaiterConfig={
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	784 'Delay': 5,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	785 })
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	786 print('%s is stopped' % instance.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	787
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	788 image = instance.create_image(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	789 Name=name,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	790 Description=description,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	791 )
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	792
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	793 image.create_tags(Tags=[
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	794 {
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	795 'Key': 'HGIMAGEFINGERPRINT',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	796 'Value': fingerprint,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	797 },
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	798 ])
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	799
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	800 print('waiting for image %s' % image.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	801
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	802 ec2client.get_waiter('image_available').wait(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	803 ImageIds=[image.id],
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	804 )
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	805
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	806 print('image %s available as %s' % (image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	807
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	808 return image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	809
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	810
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	811 def ensure_linux_dev_ami(c: AWSConnection, distro='debian10', prefix='hg-'):
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	812 """Ensures a Linux development AMI is available and up-to-date.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	813
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	814 Returns an ``ec2.Image`` of either an existing AMI or a newly-built one.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	815 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	816 ec2client = c.ec2client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	817 ec2resource = c.ec2resource
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	818
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	819 name = '%s%s-%s' % (prefix, 'linux-dev', distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	820
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	821 if distro == 'debian9':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	822 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	823 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	824 DEBIAN_ACCOUNT_ID,
43019 6952d42f9158 automation: use latest AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 43018 diff changeset	825 'debian-stretch-hvm-x86_64-gp2-2019-09-08-17994',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	826 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	827 ssh_username = 'admin'
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	828 elif distro == 'debian10':
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	829 image = find_image(
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	830 ec2resource,
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	831 DEBIAN_ACCOUNT_ID_2,
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	832 'debian-10-amd64-20190909-10',
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	833 )
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	834 ssh_username = 'admin'
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	835 elif distro == 'ubuntu18.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	836 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	837 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	838 UBUNTU_ACCOUNT_ID,
43019 6952d42f9158 automation: use latest AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 43018 diff changeset	839 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190918',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	840 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	841 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	842 elif distro == 'ubuntu19.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	843 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	844 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	845 UBUNTU_ACCOUNT_ID,
43019 6952d42f9158 automation: use latest AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 43018 diff changeset	846 'ubuntu/images/hvm-ssd/ubuntu-disco-19.04-amd64-server-20190918',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	847 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	848 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	849 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	850 raise ValueError('unsupported Linux distro: %s' % distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	851
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	852 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	853 'BlockDeviceMappings': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	854 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	855 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	856 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	857 'DeleteOnTermination': True,
43018 92c8bae84e7a automation: increase size of Linux AMI build volume Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	858 'VolumeSize': 10,
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	859 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	860 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	861 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	862 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	863 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	864 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	865 'InstanceInitiatedShutdownBehavior': 'stop',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	866 # 8 VCPUs for compiling Python.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	867 'InstanceType': 't3.2xlarge',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	868 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	869 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	870 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	871 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	872 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	873
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	874 requirements2_path = (pathlib.Path(__file__).parent.parent /
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	875 'linux-requirements-py2.txt')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	876 requirements3_path = (pathlib.Path(__file__).parent.parent /
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	877 'linux-requirements-py3.txt')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	878 with requirements2_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	879 requirements2 = fh.read()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	880 with requirements3_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	881 requirements3 = fh.read()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	882
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	883 # Compute a deterministic fingerprint to determine whether image needs to
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	884 # be regenerated.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	885 fingerprint = resolve_fingerprint({
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	886 'instance_config': config,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	887 'bootstrap_script': BOOTSTRAP_DEBIAN,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	888 'requirements_py2': requirements2,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	889 'requirements_py3': requirements3,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	890 })
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	891
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	892 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	893
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	894 if existing_image:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	895 return existing_image
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	896
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	897 print('no suitable %s image found; creating one...' % name)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	898
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	899 with temporary_ec2_instances(ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	900 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	901
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	902 instance = instances[0]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	903
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	904 client = wait_for_ssh(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	905 instance.public_ip_address, 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	906 username=ssh_username,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	907 key_filename=str(c.key_pair_path_private('automation')))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	908
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	909 home = '/home/%s' % ssh_username
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	910
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	911 with client:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	912 print('connecting to SSH server')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	913 sftp = client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	914
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	915 print('uploading bootstrap files')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	916 with sftp.open('%s/bootstrap' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	917 fh.write(BOOTSTRAP_DEBIAN)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	918 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	919
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	920 with sftp.open('%s/requirements-py2.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	921 fh.write(requirements2)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	922 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	923
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	924 with sftp.open('%s/requirements-py3.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	925 fh.write(requirements3)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	926 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	927
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	928 print('executing bootstrap')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	929 chan, stdin, stdout = ssh_exec_command(client,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	930 '%s/bootstrap' % home)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	931 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	932
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	933 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	934 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	935
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	936 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	937 if res:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	938 raise Exception('non-0 exit from bootstrap: %d' % res)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	939
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	940 print('bootstrap completed; stopping %s to create %s' % (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	941 instance.id, name))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	942
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	943 return create_ami_from_instance(ec2client, instance, name,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	944 'Mercurial Linux development environment',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	945 fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	946
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	947
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	948 @contextlib.contextmanager
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	949 def temporary_linux_dev_instances(c: AWSConnection, image, instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	950 prefix='hg-', ensure_extra_volume=False):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	951 """Create temporary Linux development EC2 instances.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	952
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	953 Context manager resolves to a list of ``ec2.Instance`` that were created
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	954 and are running.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	955
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	956 ``ensure_extra_volume`` can be set to ``True`` to require that instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	957 have a 2nd storage volume available other than the primary AMI volume.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	958 For instance types with instance storage, this does nothing special.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	959 But for instance types without instance storage, an additional EBS volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	960 will be added to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	961
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	962 Instances have an ``ssh_client`` attribute containing a paramiko SSHClient
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	963 instance bound to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	964
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	965 Instances have an ``ssh_private_key_path`` attributing containing the
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	966 str path to the SSH private key to connect to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	967 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	968
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	969 block_device_mappings = [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	970 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	971 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	972 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	973 'DeleteOnTermination': True,
42719 3e3fb15bfeea automation: increase root volume size on Linux Gregory Szorc <gregory.szorc@gmail.com> parents: 42648 diff changeset	974 'VolumeSize': 12,
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	975 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	976 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	977 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	978 ]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	979
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	980 # This is not an exhaustive list of instance types having instance storage.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	981 # But
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	982 if (ensure_extra_volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	983 and not instance_type.startswith(tuple(INSTANCE_TYPES_WITH_STORAGE))):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	984 main_device = block_device_mappings[0]['DeviceName']
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	985
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	986 if main_device == 'xvda':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	987 second_device = 'xvdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	988 elif main_device == '/dev/sda1':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	989 second_device = '/dev/sdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	990 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	991 raise ValueError('unhandled primary EBS device name: %s' %
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	992 main_device)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	993
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	994 block_device_mappings.append({
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	995 'DeviceName': second_device,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	996 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	997 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	998 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	999 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1000 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1001 })
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1002
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1003 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1004 'BlockDeviceMappings': block_device_mappings,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1005 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1006 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1007 'InstanceInitiatedShutdownBehavior': 'terminate',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1008 'InstanceType': instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1009 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1010 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1011 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1012 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1013 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1014
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1015 with temporary_ec2_instances(c.ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1016 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1017
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1018 ssh_private_key_path = str(c.key_pair_path_private('automation'))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1019
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1020 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1021 client = wait_for_ssh(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1022 instance.public_ip_address, 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1023 username='hg',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1024 key_filename=ssh_private_key_path)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1025
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1026 instance.ssh_client = client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1027 instance.ssh_private_key_path = ssh_private_key_path
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1028
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1029 try:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1030 yield instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1031 finally:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1032 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1033 instance.ssh_client.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1034
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1035
42648 d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1036 def ensure_windows_dev_ami(c: AWSConnection, prefix='hg-',
d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1037 base_image_name=WINDOWS_BASE_IMAGE_NAME):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1038 """Ensure Windows Development AMI is available and up-to-date.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1039
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1040 If necessary, a modern AMI will be built by starting a temporary EC2
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1041 instance and bootstrapping it.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1042
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1043 Obsolete AMIs will be deleted so there is only a single AMI having the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1044 desired name.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1045
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1046 Returns an ``ec2.Image`` of either an existing AMI or a newly-built
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1047 one.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1048 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1049 ec2client = c.ec2client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1050 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1051 ssmclient = c.session.client('ssm')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1052
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1053 name = '%s%s' % (prefix, 'windows-dev')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1054
42648 d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1055 image = find_image(ec2resource, AMAZON_ACCOUNT_ID, base_image_name)
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1056
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1057 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1058 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1059 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1060 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1061 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1062 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1063 'VolumeSize': 32,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1064 'VolumeType': 'gp2',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1065 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1066 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1067 ],
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1068 'ImageId': image.id,
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1069 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1070 'InstanceType': 't3.medium',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1071 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1072 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1073 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1074 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1075 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1076
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1077 commands = [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1078 # Need to start the service so sshd_config is generated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1079 'Start-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1080 'Write-Output "modifying sshd_config"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1081 r'$content = Get-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1082 '$content = $content -replace "Match Group administrators","" -replace "AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys",""',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1083 r'$content \| Set-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1084 'Import-Module OpenSSHUtils',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1085 r'Repair-SshdConfigPermission C:\ProgramData\ssh\sshd_config -Confirm:$false',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1086 'Restart-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1087 'Write-Output "installing OpenSSL client"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1088 'Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1089 'Set-Service -Name sshd -StartupType "Automatic"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1090 'Write-Output "OpenSSH server running"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1091 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1092
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1093 with INSTALL_WINDOWS_DEPENDENCIES.open('r', encoding='utf-8') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1094 commands.extend(l.rstrip() for l in fh)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1095
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1096 # Disable Windows Defender when bootstrapping because it just slows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1097 # things down.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1098 commands.insert(0, 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1099 commands.append('Set-MpPreference -DisableRealtimeMonitoring $false')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1100
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1101 # Compute a deterministic fingerprint to determine whether image needs
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1102 # to be regenerated.
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1103 fingerprint = resolve_fingerprint({
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1104 'instance_config': config,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1105 'user_data': WINDOWS_USER_DATA,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1106 'initial_bootstrap': WINDOWS_BOOTSTRAP_POWERSHELL,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1107 'bootstrap_commands': commands,
42648 d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1108 'base_image_name': base_image_name,
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1109 })
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1110
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1111 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1112
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1113 if existing_image:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1114 return existing_image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1115
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1116 print('no suitable Windows development image found; creating one...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1117
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1118 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1119 assert len(instances) == 1
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1120 instance = instances[0]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1121
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1122 wait_for_ssm(ssmclient, [instance])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1123
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1124 # On first boot, install various Windows updates.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1125 # We would ideally use PowerShell Remoting for this. However, there are
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1126 # trust issues that make it difficult to invoke Windows Update
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1127 # remotely. So we use SSM, which has a mechanism for running Windows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1128 # Update.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1129 print('installing Windows features...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1130 run_ssm_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1131 ssmclient,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1132 [instance],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1133 'AWS-RunPowerShellScript',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1134 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1135 'commands': WINDOWS_BOOTSTRAP_POWERSHELL.split('\n'),
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1136 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1137 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1138
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1139 # Reboot so all updates are fully applied.
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1140 #
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1141 # We don't use instance.reboot() here because it is asynchronous and
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1142 # we don't know when exactly the instance has rebooted. It could take
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1143 # a while to stop and we may start trying to interact with the instance
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1144 # before it has rebooted.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1145 print('rebooting instance %s' % instance.id)
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1146 instance.stop()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1147 ec2client.get_waiter('instance_stopped').wait(
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1148 InstanceIds=[instance.id],
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1149 WaiterConfig={
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1150 'Delay': 5,
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1151 })
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1152
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1153 instance.start()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1154 wait_for_ip_addresses([instance])
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1155
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1156 # There is a race condition here between the User Data PS script running
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1157 # and us connecting to WinRM. This can manifest as
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1158 # "AuthorizationManager check failed" failures during run_powershell().
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1159 # TODO figure out a workaround.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1160
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1161 print('waiting for Windows Remote Management to come back...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1162 client = wait_for_winrm(instance.public_ip_address, 'Administrator',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1163 c.automation.default_password())
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1164 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1165 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1166
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1167 print('bootstrapping instance...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1168 run_powershell(instance.winrm_client, '\n'.join(commands))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1169
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1170 print('bootstrap completed; stopping %s to create image' % instance.id)
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1171 return create_ami_from_instance(ec2client, instance, name,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1172 'Mercurial Windows development environment',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1173 fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1174
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1175
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1176 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1177 def temporary_windows_dev_instances(c: AWSConnection, image, instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1178 prefix='hg-', disable_antivirus=False):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1179 """Create a temporary Windows development EC2 instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1180
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1181 Context manager resolves to the list of ``EC2.Instance`` that were created.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1182 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1183 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1184 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1185 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1186 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1187 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1188 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1189 'VolumeSize': 32,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1190 'VolumeType': 'gp2',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1191 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1192 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1193 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1194 'ImageId': image.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1195 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1196 'InstanceType': instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1197 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1198 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1199 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1200 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1201 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1202
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1203 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1204 if disable_antivirus:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1205 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1206 run_powershell(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1207 instance.winrm_client,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1208 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1209
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1210 yield instances

Mercurial > public > mercurial-scm > hg-stable

annotate contrib/automation/hgautomation/aws.py @ 43020:d1d919f679f7