mercurial-scm/hg: contrib/automation/hgautomation/aws.py annotate

annotate contrib/automation/hgautomation/aws.py @ 42648:d80edcb0b30c stable

automation: make Windows base image name configurable Since automation broke in the middle of the 5.0 release cycle, there's a good chance it will break again in the future. While a robust solution might be to search for all available images and choose the newest one, it does seem useful to be able to explicitly choose the name of the image to find and use so users can opt in to using a different image. This commit implements that functionality. Differential Revision: https://phab.mercurial-scm.org/D6673

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Mon, 22 Jul 2019 19:06:20 -0700
parents	8804aa6c07a0
children	3e3fb15bfeea

rev	line source
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # aws.py - Automation code for Amazon Web Services
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 # no-check-code because Python 3 native.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10 import contextlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11 import copy
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12 import hashlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13 import json
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 import os
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 import pathlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	16 import subprocess
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 import time
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 import boto3
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	20 import botocore.exceptions
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	21
42285 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	22 from .linux import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	23 BOOTSTRAP_DEBIAN,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	24 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	25 from .ssh import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	26 exec_command as ssh_exec_command,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	27 wait_for_ssh,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	28 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 from .winrm import (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 run_powershell,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 wait_for_winrm,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	34
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	35 SOURCE_ROOT = pathlib.Path(os.path.abspath(__file__)).parent.parent.parent.parent
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	37 INSTALL_WINDOWS_DEPENDENCIES = (SOURCE_ROOT / 'contrib' /
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	38 'install-windows-dependencies.ps1')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	39
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	40
42285 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	41 INSTANCE_TYPES_WITH_STORAGE = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	42 'c5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	43 'd2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	44 'h1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	45 'i3',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	46 'm5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	47 'm5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	48 'r5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	49 'r5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	50 'x1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	51 'z1d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	52 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	53
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	54
42647 8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	55 AMAZON_ACCOUNT_ID = '801119661308'
42285 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	56 DEBIAN_ACCOUNT_ID = '379101102735'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	57 UBUNTU_ACCOUNT_ID = '099720109477'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	58
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	59
42647 8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	60 WINDOWS_BASE_IMAGE_NAME = 'Windows_Server-2019-English-Full-Base-2019.07.12'
8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	61
8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	62
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	63 KEY_PAIRS = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	64 'automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	65 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	66
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68 SECURITY_GROUPS = {
42285 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	69 'linux-dev-1': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	70 'description': 'Mercurial Linux instances that perform build/test automation',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	71 'ingress': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	72 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	73 'FromPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	74 'ToPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	75 'IpProtocol': 'tcp',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	76 'IpRanges': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	77 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	78 'CidrIp': '0.0.0.0/0',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	79 'Description': 'SSH from entire Internet',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	80 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	81 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	82 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	83 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	84 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	85 'windows-dev-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 'description': 'Mercurial Windows instances that perform build automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 'ingress': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 'FromPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 'ToPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 'Description': 'SSH from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100 'FromPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 'ToPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 'Description': 'RDP from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 'FromPort': 5985,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 'ToPort': 5986,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118 'Description': 'PowerShell Remoting (Windows Remote Management)',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	126
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127 IAM_ROLES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 'ephemeral-ec2-role-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 'description': 'Mercurial temporary EC2 instances',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130 'policy_arns': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137 ASSUME_ROLE_POLICY_DOCUMENT = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 "Version": "2012-10-17",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	140 "Statement": [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	142 "Effect": "Allow",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 "Principal": {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 "Service": "ec2.amazonaws.com"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 "Action": "sts:AssumeRole"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	149 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150 '''.strip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	153 IAM_INSTANCE_PROFILES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	154 'ephemeral-ec2-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	155 'roles': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	156 'ephemeral-ec2-role-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	157 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	158 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	159 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	161
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	162 # User Data for Windows EC2 instance. Mainly used to set the password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	163 # and configure WinRM.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164 # Inspired by the User Data script used by Packer
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	165 # (from https://www.packer.io/intro/getting-started/build-image.html).
42064 0e9066db5e44 automation: use raw strings when there are backslashes Gregory Szorc <gregory.szorc@gmail.com> parents: 42024 diff changeset	166 WINDOWS_USER_DATA = r'''
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	167 <powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169 # TODO enable this once we figure out what is failing.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170 #$ErrorActionPreference = "stop"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172 # Set administrator password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	173 net user Administrator "%s"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	174 wmic useraccount where "name='Administrator'" set PasswordExpires=FALSE
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	175
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176 # First, make sure WinRM can't be connected to
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	177 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new enable=yes action=block
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	178
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	179 # Delete any existing WinRM listeners
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	180 winrm delete winrm/config/listener?Address=*+Transport=HTTP 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	181 winrm delete winrm/config/listener?Address=*+Transport=HTTPS 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	183 # Create a new WinRM listener and configure
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	184 winrm create winrm/config/listener?Address=*+Transport=HTTP
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	185 winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="0"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	186 winrm set winrm/config '@{MaxTimeoutms="7200000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	187 winrm set winrm/config/service '@{AllowUnencrypted="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	188 winrm set winrm/config/service '@{MaxConcurrentOperationsPerUser="12000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	189 winrm set winrm/config/service/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	190 winrm set winrm/config/client/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	191
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	192 # Configure UAC to allow privilege elevation in remote shells
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	193 $Key = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	194 $Setting = 'LocalAccountTokenFilterPolicy'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	195 Set-ItemProperty -Path $Key -Name $Setting -Value 1 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	196
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	197 # Configure and restart the WinRM Service; Enable the required firewall exception
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	198 Stop-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	199 Set-Service -Name WinRM -StartupType Automatic
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	200 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new action=allow localip=any remoteip=any
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201 Start-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	202
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	203 # Disable firewall on private network interfaces so prompts don't appear.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204 Set-NetFirewallProfile -Name private -Enabled false
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	205 </powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	206 '''.lstrip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	207
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	208
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209 WINDOWS_BOOTSTRAP_POWERSHELL = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	210 Write-Output "installing PowerShell dependencies"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212 Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213 Install-Module -Name OpenSSHUtils -RequiredVersion 0.0.2.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	215 Write-Output "installing OpenSSL server"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	216 Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	217 # Various tools will attempt to use older versions of .NET. So we enable
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	218 # the feature that provides them so it doesn't have to be auto-enabled
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219 # later.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220 Write-Output "enabling .NET Framework feature"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221 Install-WindowsFeature -Name Net-Framework-Core
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222 '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	223
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	224
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	225 class AWSConnection:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	226 """Manages the state of a connection with AWS."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	227
42277 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42276 diff changeset	228 def __init__(self, automation, region: str, ensure_ec2_state: bool=True):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	229 self.automation = automation
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	230 self.local_state_path = automation.state_path
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	231
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	232 self.prefix = 'hg-'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 self.session = boto3.session.Session(region_name=region)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235 self.ec2client = self.session.client('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236 self.ec2resource = self.session.resource('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	237 self.iamclient = self.session.client('iam')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	238 self.iamresource = self.session.resource('iam')
42277 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42276 diff changeset	239 self.security_groups = {}
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	240
42277 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42276 diff changeset	241 if ensure_ec2_state:
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42276 diff changeset	242 ensure_key_pairs(automation.state_path, self.ec2resource)
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42276 diff changeset	243 self.security_groups = ensure_security_groups(self.ec2resource)
42278 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	244 ensure_iam_state(self.iamclient, self.iamresource)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	245
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	246 def key_pair_path_private(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	247 """Path to a key pair private key file."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	248 return self.local_state_path / 'keys' / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	249
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	250 def key_pair_path_public(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	251 return self.local_state_path / 'keys' / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	252
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	253
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	254 def rsa_key_fingerprint(p: pathlib.Path):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	255 """Compute the fingerprint of an RSA private key."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	256
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257 # TODO use rsa package.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 res = subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259 ['openssl', 'pkcs8', '-in', str(p), '-nocrypt', '-topk8',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	260 '-outform', 'DER'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	261 capture_output=True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	262 check=True)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	263
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	264 sha1 = hashlib.sha1(res.stdout).hexdigest()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	265 return ':'.join(a + b for a, b in zip(sha1[::2], sha1[1::2]))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	266
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	267
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	268 def ensure_key_pairs(state_path: pathlib.Path, ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	269 remote_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	271 for kpi in ec2resource.key_pairs.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 if kpi.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 remote_existing[kpi.name[len(prefix):]] = kpi.key_fingerprint
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275 # Validate that we have these keys locally.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 key_path = state_path / 'keys'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 key_path.mkdir(exist_ok=True, mode=0o700)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279 def remove_remote(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 print('deleting key pair %s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 key = ec2resource.KeyPair(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 key.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284 def remove_local(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288 print('removing %s' % pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 print('removing %s' % priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	293 local_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 for f in sorted(os.listdir(key_path)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 if not f.startswith('keypair-') or not f.endswith('.pub'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299 name = f[len('keypair-'):-len('.pub')]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301 pub_full = key_path / f
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304 with open(pub_full, 'r', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305 data = fh.read()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307 if not data.startswith('ssh-rsa '):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 print('unexpected format for key pair file: %s; removing' %
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	309 pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314 local_existing[name] = rsa_key_fingerprint(priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	316 for name in sorted(set(remote_existing) \| set(local_existing)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 if name not in local_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319 print('remote key %s does not exist locally' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 remove_remote(actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323 elif name not in remote_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324 print('local key %s does not exist remotely' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	325 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328 elif remote_existing[name] != local_existing[name]:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329 print('key fingerprint mismatch for %s; '
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	330 'removing from local and remote' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	332 remove_remote('%s%s' % (prefix, name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	333 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	334 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	335
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	336 missing = KEY_PAIRS - set(remote_existing)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	337
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	338 for name in sorted(missing):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	339 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	340 print('creating key pair %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	341
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345 kp = ec2resource.create_key_pair(KeyName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	347 with priv_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348 fh.write(kp.key_material)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	349 fh.write('\n')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	350
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	351 priv_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	352
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	353 # SSH public key can be extracted via `ssh-keygen`.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	354 with pub_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	355 subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	356 ['ssh-keygen', '-y', '-f', str(priv_full)],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357 stdout=fh,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358 check=True)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360 pub_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	361
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	362
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	363 def delete_instance_profile(profile):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	364 for role in profile.roles:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365 print('removing role %s from instance profile %s' % (role.name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	366 profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	367 profile.remove_role(RoleName=role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	368
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	369 print('deleting instance profile %s' % profile.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370 profile.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372
42278 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	373 def ensure_iam_state(iamclient, iamresource, prefix='hg-'):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	374 """Ensure IAM state is in sync with our canonical definition."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	375
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376 remote_profiles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380 remote_profiles[profile.name[len(prefix):]] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382 for name in sorted(set(remote_profiles) - set(IAM_INSTANCE_PROFILES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383 delete_instance_profile(remote_profiles[name])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384 del remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	386 remote_roles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	387
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	388 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	389 if role.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	390 remote_roles[role.name[len(prefix):]] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	391
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	392 for name in sorted(set(remote_roles) - set(IAM_ROLES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	393 role = remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	394
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	395 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397 del remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399 # We've purged remote state that doesn't belong. Create missing
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	400 # instance profiles and roles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	401 for name in sorted(set(IAM_INSTANCE_PROFILES) - set(remote_profiles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	402 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	403 print('creating IAM instance profile %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	405 profile = iamresource.create_instance_profile(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	406 InstanceProfileName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	407 remote_profiles[name] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	408
42278 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	409 waiter = iamclient.get_waiter('instance_profile_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	410 waiter.wait(InstanceProfileName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	411 print('IAM instance profile %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	412
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	413 for name in sorted(set(IAM_ROLES) - set(remote_roles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	414 entry = IAM_ROLES[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	415
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	416 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	417 print('creating IAM role %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	418
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	419 role = iamresource.create_role(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	420 RoleName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	421 Description=entry['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	422 AssumeRolePolicyDocument=ASSUME_ROLE_POLICY_DOCUMENT,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	423 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	424
42278 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	425 waiter = iamclient.get_waiter('role_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	426 waiter.wait(RoleName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	427 print('IAM role %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42277 diff changeset	428
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	429 remote_roles[name] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431 for arn in entry['policy_arns']:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 print('attaching policy %s to %s' % (arn, role.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433 role.attach_policy(PolicyArn=arn)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	435 # Now reconcile state of profiles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	436 for name, meta in sorted(IAM_INSTANCE_PROFILES.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	437 profile = remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438 wanted = {'%s%s' % (prefix, role) for role in meta['roles']}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439 have = {role.name for role in profile.roles}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441 for role in sorted(have - wanted):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442 print('removing role %s from %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	443 profile.remove_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	444
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445 for role in sorted(wanted - have):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	446 print('adding role %s to %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	447 profile.add_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	448
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	449
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	450 def find_image(ec2resource, owner_id, name):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	451 """Find an AMI by its owner ID and name."""
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	452
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	453 images = ec2resource.images.filter(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	454 Filters=[
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	455 {
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	456 'Name': 'owner-id',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	457 'Values': [owner_id],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	458 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	460 'Name': 'state',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461 'Values': ['available'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	462 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	463 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	464 'Name': 'image-type',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	465 'Values': ['machine'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	466 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	467 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	468 'Name': 'name',
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	469 'Values': [name],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	470 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	471 ])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	472
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	473 for image in images:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	474 return image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	475
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	476 raise Exception('unable to find image for %s' % name)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	477
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	478
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	479 def ensure_security_groups(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	480 """Ensure all necessary Mercurial security groups are present.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	481
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	482 All security groups are prefixed with ``hg-`` by default. Any security
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	483 groups having this prefix but aren't in our list are deleted.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	484 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	485 existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	486
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	487 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	488 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	489 existing[group.group_name[len(prefix):]] = group
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	490
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	491 purge = set(existing) - set(SECURITY_GROUPS)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	492
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	493 for name in sorted(purge):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	494 group = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	495 print('removing legacy security group: %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	496 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	497
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	498 security_groups = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	499
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	500 for name, group in sorted(SECURITY_GROUPS.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	501 if name in existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	502 security_groups[name] = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	503 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	504
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	505 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	506 print('adding security group %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	507
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	508 group_res = ec2resource.create_security_group(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	509 Description=group['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	510 GroupName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	512
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	513 group_res.authorize_ingress(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	514 IpPermissions=group['ingress'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	515 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	516
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	517 security_groups[name] = group_res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	518
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	519 return security_groups
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	520
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	521
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	522 def terminate_ec2_instances(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	523 """Terminate all EC2 instances managed by us."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	524 waiting = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	525
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	526 for instance in ec2resource.instances.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	527 if instance.state['Name'] == 'terminated':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	528 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	529
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	530 for tag in instance.tags or []:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	531 if tag['Key'] == 'Name' and tag['Value'].startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	532 print('terminating %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	533 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	534 waiting.append(instance)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	535
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	536 for instance in waiting:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	537 instance.wait_until_terminated()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	538
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	539
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	540 def remove_resources(c, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	541 """Purge all of our resources in this EC2 region."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543 iamresource = c.iamresource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545 terminate_ec2_instances(ec2resource, prefix=prefix)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546
42275 730edbd836d8 automation: only iterate over our AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 42064 diff changeset	547 for image in ec2resource.images.filter(Owners=['self']):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	548 if image.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 remove_ami(ec2resource, image)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	552 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	553 print('removing security group %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	554 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	555
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	556 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	557 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	558 delete_instance_profile(profile)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	559
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	560 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	561 if role.name.startswith(prefix):
42276 fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42275 diff changeset	562 for p in role.attached_policies.all():
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42275 diff changeset	563 print('detaching policy %s from %s' % (p.arn, role.name))
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42275 diff changeset	564 role.detach_policy(PolicyArn=p.arn)
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42275 diff changeset	565
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	566 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	568
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	570 def wait_for_ip_addresses(instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571 """Wait for the public IP addresses of an iterable of instances."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	574 if not instance.public_ip_address:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	575 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	576 instance.reload()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	577 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	578
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579 print('public IP address for %s: %s' % (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	580 instance.id, instance.public_ip_address))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	581 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	582
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	583
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	584 def remove_ami(ec2resource, image):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	585 """Remove an AMI and its underlying snapshots."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586 snapshots = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	587
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	588 for device in image.block_device_mappings:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	589 if 'Ebs' in device:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	590 snapshots.append(ec2resource.Snapshot(device['Ebs']['SnapshotId']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	591
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	592 print('deregistering %s' % image.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	593 image.deregister()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	594
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595 for snapshot in snapshots:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	596 print('deleting snapshot %s' % snapshot.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	597 snapshot.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	598
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	599
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	600 def wait_for_ssm(ssmclient, instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	601 """Wait for SSM to come online for an iterable of instance IDs."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	602 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	603 res = ssmclient.describe_instance_information(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	604 Filters=[
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	605 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	606 'Key': 'InstanceIds',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	607 'Values': [i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	608 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	609 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	610 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	611
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	612 available = len(res['InstanceInformationList'])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	613 wanted = len(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	614
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	615 print('%d/%d instances available in SSM' % (available, wanted))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	616
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	617 if available == wanted:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	618 return
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	619
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	620 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	621
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	622
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	623 def run_ssm_command(ssmclient, instances, document_name, parameters):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	624 """Run a PowerShell script on an EC2 instance."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	625
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	626 res = ssmclient.send_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	627 InstanceIds=[i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	628 DocumentName=document_name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	629 Parameters=parameters,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	630 CloudWatchOutputConfig={
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	631 'CloudWatchOutputEnabled': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	632 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	633 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	634
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	635 command_id = res['Command']['CommandId']
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	636
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	637 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	638 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	639 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	640 res = ssmclient.get_command_invocation(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	641 CommandId=command_id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	642 InstanceId=instance.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	643 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	644 except botocore.exceptions.ClientError as e:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	645 if e.response['Error']['Code'] == 'InvocationDoesNotExist':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	646 print('could not find SSM command invocation; waiting')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	647 time.sleep(1)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	648 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	649 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	650 raise
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	651
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	652 if res['Status'] == 'Success':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	653 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	654 elif res['Status'] in ('Pending', 'InProgress', 'Delayed'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	655 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	656 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	657 raise Exception('command failed on %s: %s' % (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	658 instance.id, res['Status']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	659
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	660
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	661 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	662 def temporary_ec2_instances(ec2resource, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	663 """Create temporary EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	664
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	665 This is a proxy to ``ec2client.run_instances(**config)`` that takes care of
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	666 managing the lifecycle of the instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	667
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	668 When the context manager exits, the instances are terminated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	669
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	670 The context manager evaluates to the list of data structures
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	671 describing each created instance. The instances may not be available
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	672 for work immediately: it is up to the caller to wait for the instance
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	673 to start responding.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	674 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	675
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	676 ids = None
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	677
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	678 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	679 res = ec2resource.create_instances(**config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	680
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	681 ids = [i.id for i in res]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	682 print('started instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	683
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	684 yield res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	685 finally:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	686 if ids:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	687 print('terminating instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	688 for instance in res:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	689 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	690 print('terminated %d instances' % len(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	691
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	692
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	693 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	694 def create_temp_windows_ec2_instances(c: AWSConnection, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	695 """Create temporary Windows EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	696
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	697 This is a higher-level wrapper around ``create_temp_ec2_instances()`` that
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	698 configures the Windows instance for Windows Remote Management. The emitted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	699 instances will have a ``winrm_client`` attribute containing a
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	700 ``pypsrp.client.Client`` instance bound to the instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	701 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	702 if 'IamInstanceProfile' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	703 raise ValueError('IamInstanceProfile cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	704 if 'UserData' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	705 raise ValueError('UserData cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	706
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	707 password = c.automation.default_password()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	708
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	709 config = copy.deepcopy(config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	710 config['IamInstanceProfile'] = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	711 'Name': 'hg-ephemeral-ec2-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	712 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	713 config.setdefault('TagSpecifications', []).append({
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	714 'ResourceType': 'instance',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	715 'Tags': [{'Key': 'Name', 'Value': 'hg-temp-windows'}],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	716 })
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	717 config['UserData'] = WINDOWS_USER_DATA % password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	718
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	719 with temporary_ec2_instances(c.ec2resource, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	720 wait_for_ip_addresses(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	721
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	722 print('waiting for Windows Remote Management service...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	724 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	725 client = wait_for_winrm(instance.public_ip_address, 'Administrator', password)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	726 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	727 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	728
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	729 yield instances
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	730
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	731
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	732 def resolve_fingerprint(fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	733 fingerprint = json.dumps(fingerprint, sort_keys=True)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	734 return hashlib.sha256(fingerprint.encode('utf-8')).hexdigest()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	735
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	736
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	737 def find_and_reconcile_image(ec2resource, name, fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	738 """Attempt to find an existing EC2 AMI with a name and fingerprint.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	739
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	740 If an image with the specified fingerprint is found, it is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	741 Otherwise None is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	742
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	743 Existing images for the specified name that don't have the specified
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	744 fingerprint or are missing required metadata or deleted.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	745 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	746 # Find existing AMIs with this name and delete the ones that are invalid.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	747 # Store a reference to a good image so it can be returned one the
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	748 # image state is reconciled.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	749 images = ec2resource.images.filter(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	750 Filters=[{'Name': 'name', 'Values': [name]}])
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	751
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	752 existing_image = None
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	753
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	754 for image in images:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	755 if image.tags is None:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	756 print('image %s for %s lacks required tags; removing' % (
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	757 image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	758 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	759 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	760 tags = {t['Key']: t['Value'] for t in image.tags}
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	761
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	762 if tags.get('HGIMAGEFINGERPRINT') == fingerprint:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	763 existing_image = image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	764 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	765 print('image %s for %s has wrong fingerprint; removing' % (
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	766 image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	767 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	768
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	769 return existing_image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	770
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	771
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	772 def create_ami_from_instance(ec2client, instance, name, description,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	773 fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	774 """Create an AMI from a running instance.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	775
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	776 Returns the ``ec2resource.Image`` representing the created AMI.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	777 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	778 instance.stop()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	779
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	780 ec2client.get_waiter('instance_stopped').wait(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	781 InstanceIds=[instance.id],
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	782 WaiterConfig={
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	783 'Delay': 5,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	784 })
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	785 print('%s is stopped' % instance.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	786
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	787 image = instance.create_image(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	788 Name=name,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	789 Description=description,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	790 )
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	791
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	792 image.create_tags(Tags=[
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	793 {
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	794 'Key': 'HGIMAGEFINGERPRINT',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	795 'Value': fingerprint,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	796 },
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	797 ])
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	798
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	799 print('waiting for image %s' % image.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	800
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	801 ec2client.get_waiter('image_available').wait(
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	802 ImageIds=[image.id],
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	803 )
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	804
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	805 print('image %s available as %s' % (image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	806
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	807 return image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	808
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	809
42285 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	810 def ensure_linux_dev_ami(c: AWSConnection, distro='debian9', prefix='hg-'):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	811 """Ensures a Linux development AMI is available and up-to-date.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	812
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	813 Returns an ``ec2.Image`` of either an existing AMI or a newly-built one.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	814 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	815 ec2client = c.ec2client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	816 ec2resource = c.ec2resource
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	817
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	818 name = '%s%s-%s' % (prefix, 'linux-dev', distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	819
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	820 if distro == 'debian9':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	821 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	822 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	823 DEBIAN_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	824 'debian-stretch-hvm-x86_64-gp2-2019-02-19-26620',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	825 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	826 ssh_username = 'admin'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	827 elif distro == 'ubuntu18.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	828 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	829 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	830 UBUNTU_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	831 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190403',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	832 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	833 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	834 elif distro == 'ubuntu18.10':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	835 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	836 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	837 UBUNTU_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	838 'ubuntu/images/hvm-ssd/ubuntu-cosmic-18.10-amd64-server-20190402',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	839 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	840 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	841 elif distro == 'ubuntu19.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	842 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	843 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	844 UBUNTU_ACCOUNT_ID,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	845 'ubuntu/images/hvm-ssd/ubuntu-disco-19.04-amd64-server-20190417',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	846 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	847 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	848 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	849 raise ValueError('unsupported Linux distro: %s' % distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	850
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	851 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	852 'BlockDeviceMappings': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	853 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	854 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	855 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	856 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	857 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	858 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	859 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	860 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	861 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	862 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	863 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	864 'InstanceInitiatedShutdownBehavior': 'stop',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	865 # 8 VCPUs for compiling Python.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	866 'InstanceType': 't3.2xlarge',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	867 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	868 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	869 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	870 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	871 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	872
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	873 requirements2_path = (pathlib.Path(__file__).parent.parent /
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	874 'linux-requirements-py2.txt')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	875 requirements3_path = (pathlib.Path(__file__).parent.parent /
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	876 'linux-requirements-py3.txt')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	877 with requirements2_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	878 requirements2 = fh.read()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	879 with requirements3_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	880 requirements3 = fh.read()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	881
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	882 # Compute a deterministic fingerprint to determine whether image needs to
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	883 # be regenerated.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	884 fingerprint = resolve_fingerprint({
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	885 'instance_config': config,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	886 'bootstrap_script': BOOTSTRAP_DEBIAN,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	887 'requirements_py2': requirements2,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	888 'requirements_py3': requirements3,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	889 })
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	890
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	891 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	892
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	893 if existing_image:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	894 return existing_image
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	895
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	896 print('no suitable %s image found; creating one...' % name)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	897
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	898 with temporary_ec2_instances(ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	899 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	900
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	901 instance = instances[0]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	902
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	903 client = wait_for_ssh(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	904 instance.public_ip_address, 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	905 username=ssh_username,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	906 key_filename=str(c.key_pair_path_private('automation')))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	907
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	908 home = '/home/%s' % ssh_username
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	909
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	910 with client:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	911 print('connecting to SSH server')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	912 sftp = client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	913
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	914 print('uploading bootstrap files')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	915 with sftp.open('%s/bootstrap' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	916 fh.write(BOOTSTRAP_DEBIAN)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	917 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	918
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	919 with sftp.open('%s/requirements-py2.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	920 fh.write(requirements2)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	921 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	922
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	923 with sftp.open('%s/requirements-py3.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	924 fh.write(requirements3)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	925 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	926
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	927 print('executing bootstrap')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	928 chan, stdin, stdout = ssh_exec_command(client,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	929 '%s/bootstrap' % home)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	930 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	931
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	932 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	933 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	934
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	935 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	936 if res:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	937 raise Exception('non-0 exit from bootstrap: %d' % res)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	938
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	939 print('bootstrap completed; stopping %s to create %s' % (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	940 instance.id, name))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	941
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	942 return create_ami_from_instance(ec2client, instance, name,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	943 'Mercurial Linux development environment',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	944 fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	945
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	946
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	947 @contextlib.contextmanager
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	948 def temporary_linux_dev_instances(c: AWSConnection, image, instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	949 prefix='hg-', ensure_extra_volume=False):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	950 """Create temporary Linux development EC2 instances.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	951
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	952 Context manager resolves to a list of ``ec2.Instance`` that were created
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	953 and are running.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	954
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	955 ``ensure_extra_volume`` can be set to ``True`` to require that instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	956 have a 2nd storage volume available other than the primary AMI volume.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	957 For instance types with instance storage, this does nothing special.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	958 But for instance types without instance storage, an additional EBS volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	959 will be added to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	960
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	961 Instances have an ``ssh_client`` attribute containing a paramiko SSHClient
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	962 instance bound to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	963
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	964 Instances have an ``ssh_private_key_path`` attributing containing the
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	965 str path to the SSH private key to connect to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	966 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	967
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	968 block_device_mappings = [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	969 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	970 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	971 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	972 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	973 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	974 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	975 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	976 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	977 ]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	978
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	979 # This is not an exhaustive list of instance types having instance storage.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	980 # But
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	981 if (ensure_extra_volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	982 and not instance_type.startswith(tuple(INSTANCE_TYPES_WITH_STORAGE))):
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	983 main_device = block_device_mappings[0]['DeviceName']
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	984
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	985 if main_device == 'xvda':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	986 second_device = 'xvdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	987 elif main_device == '/dev/sda1':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	988 second_device = '/dev/sdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	989 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	990 raise ValueError('unhandled primary EBS device name: %s' %
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	991 main_device)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	992
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	993 block_device_mappings.append({
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	994 'DeviceName': second_device,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	995 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	996 'DeleteOnTermination': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	997 'VolumeSize': 8,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	998 'VolumeType': 'gp2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	999 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1000 })
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1001
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1002 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1003 'BlockDeviceMappings': block_device_mappings,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1004 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1005 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1006 'InstanceInitiatedShutdownBehavior': 'terminate',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1007 'InstanceType': instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1008 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1009 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1010 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1011 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1012 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1013
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1014 with temporary_ec2_instances(c.ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1015 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1016
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1017 ssh_private_key_path = str(c.key_pair_path_private('automation'))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1018
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1019 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1020 client = wait_for_ssh(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1021 instance.public_ip_address, 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1022 username='hg',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1023 key_filename=ssh_private_key_path)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1024
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1025 instance.ssh_client = client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1026 instance.ssh_private_key_path = ssh_private_key_path
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1027
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1028 try:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1029 yield instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1030 finally:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1031 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1032 instance.ssh_client.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1033
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42284 diff changeset	1034
42648 d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1035 def ensure_windows_dev_ami(c: AWSConnection, prefix='hg-',
d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1036 base_image_name=WINDOWS_BASE_IMAGE_NAME):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1037 """Ensure Windows Development AMI is available and up-to-date.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1038
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1039 If necessary, a modern AMI will be built by starting a temporary EC2
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1040 instance and bootstrapping it.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1041
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1042 Obsolete AMIs will be deleted so there is only a single AMI having the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1043 desired name.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1044
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1045 Returns an ``ec2.Image`` of either an existing AMI or a newly-built
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1046 one.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1047 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1048 ec2client = c.ec2client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1049 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1050 ssmclient = c.session.client('ssm')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1051
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1052 name = '%s%s' % (prefix, 'windows-dev')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1053
42648 d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1054 image = find_image(ec2resource, AMAZON_ACCOUNT_ID, base_image_name)
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1055
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1056 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1057 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1058 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1059 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1060 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1061 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1062 'VolumeSize': 32,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1063 'VolumeType': 'gp2',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1064 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1065 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1066 ],
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1067 'ImageId': image.id,
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1068 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1069 'InstanceType': 't3.medium',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1070 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1071 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1072 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1073 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1074 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1075
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1076 commands = [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1077 # Need to start the service so sshd_config is generated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1078 'Start-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1079 'Write-Output "modifying sshd_config"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1080 r'$content = Get-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1081 '$content = $content -replace "Match Group administrators","" -replace "AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys",""',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1082 r'$content \| Set-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1083 'Import-Module OpenSSHUtils',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1084 r'Repair-SshdConfigPermission C:\ProgramData\ssh\sshd_config -Confirm:$false',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1085 'Restart-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1086 'Write-Output "installing OpenSSL client"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1087 'Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1088 'Set-Service -Name sshd -StartupType "Automatic"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1089 'Write-Output "OpenSSH server running"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1090 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1091
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1092 with INSTALL_WINDOWS_DEPENDENCIES.open('r', encoding='utf-8') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1093 commands.extend(l.rstrip() for l in fh)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1094
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1095 # Disable Windows Defender when bootstrapping because it just slows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1096 # things down.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1097 commands.insert(0, 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1098 commands.append('Set-MpPreference -DisableRealtimeMonitoring $false')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1099
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1100 # Compute a deterministic fingerprint to determine whether image needs
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1101 # to be regenerated.
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1102 fingerprint = resolve_fingerprint({
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1103 'instance_config': config,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1104 'user_data': WINDOWS_USER_DATA,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1105 'initial_bootstrap': WINDOWS_BOOTSTRAP_POWERSHELL,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1106 'bootstrap_commands': commands,
42648 d80edcb0b30c automation: make Windows base image name configurable Gregory Szorc <gregory.szorc@gmail.com> parents: 42647 diff changeset	1107 'base_image_name': base_image_name,
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1108 })
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1109
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1110 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1111
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1112 if existing_image:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1113 return existing_image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1114
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1115 print('no suitable Windows development image found; creating one...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1116
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1117 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1118 assert len(instances) == 1
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1119 instance = instances[0]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1120
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1121 wait_for_ssm(ssmclient, [instance])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1122
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1123 # On first boot, install various Windows updates.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1124 # We would ideally use PowerShell Remoting for this. However, there are
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1125 # trust issues that make it difficult to invoke Windows Update
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1126 # remotely. So we use SSM, which has a mechanism for running Windows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1127 # Update.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1128 print('installing Windows features...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1129 run_ssm_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1130 ssmclient,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1131 [instance],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1132 'AWS-RunPowerShellScript',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1133 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1134 'commands': WINDOWS_BOOTSTRAP_POWERSHELL.split('\n'),
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1135 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1136 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1137
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1138 # Reboot so all updates are fully applied.
42280 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1139 #
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1140 # We don't use instance.reboot() here because it is asynchronous and
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1141 # we don't know when exactly the instance has rebooted. It could take
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1142 # a while to stop and we may start trying to interact with the instance
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1143 # before it has rebooted.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1144 print('rebooting instance %s' % instance.id)
42280 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1145 instance.stop()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1146 ec2client.get_waiter('instance_stopped').wait(
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1147 InstanceIds=[instance.id],
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1148 WaiterConfig={
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1149 'Delay': 5,
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1150 })
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1151
42280 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1152 instance.start()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1153 wait_for_ip_addresses([instance])
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1154
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1155 # There is a race condition here between the User Data PS script running
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1156 # and us connecting to WinRM. This can manifest as
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1157 # "AuthorizationManager check failed" failures during run_powershell().
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42278 diff changeset	1158 # TODO figure out a workaround.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1159
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1160 print('waiting for Windows Remote Management to come back...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1161 client = wait_for_winrm(instance.public_ip_address, 'Administrator',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1162 c.automation.default_password())
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1163 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1164 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1165
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1166 print('bootstrapping instance...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1167 run_powershell(instance.winrm_client, '\n'.join(commands))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1168
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1169 print('bootstrap completed; stopping %s to create image' % instance.id)
42284 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1170 return create_ami_from_instance(ec2client, instance, name,
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1171 'Mercurial Windows development environment',
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42280 diff changeset	1172 fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1173
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1174
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1175 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1176 def temporary_windows_dev_instances(c: AWSConnection, image, instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1177 prefix='hg-', disable_antivirus=False):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1178 """Create a temporary Windows development EC2 instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1179
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1180 Context manager resolves to the list of ``EC2.Instance`` that were created.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1181 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1182 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1183 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1184 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1185 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1186 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1187 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1188 'VolumeSize': 32,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1189 'VolumeType': 'gp2',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1190 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1191 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1192 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1193 'ImageId': image.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1194 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1195 'InstanceType': instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1196 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1197 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1198 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1199 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1200 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1201
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1202 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1203 if disable_antivirus:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1204 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1205 run_powershell(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1206 instance.winrm_client,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1207 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1208
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1209 yield instances

Mercurial > public > mercurial-scm > hg

annotate contrib/automation/hgautomation/aws.py @ 42648:d80edcb0b30c stable