mercurial-scm/hg-stable: contrib/automation/hgautomation/aws.py annotate

annotate contrib/automation/hgautomation/aws.py @ 49402:f1dcddb7f328 stable

automation: transition to Windows Server 2022 Let's keep our Windows build environment modern by upgrading to the latest OS. As part of the upgrade, we pick up a migration to EC2Launch Version 2. This has a different config mechanism. So we need to port how we manage the administrator password. As part of migrating to the new YAML/JSON config file mechanism, we move the code to the powershell script that is run when the instance first launches. This ensures that the config is retained during the reboot we perform as part of building the Windows AMI. The motivation for this is I'm currently unable to build the Windows 2019 AMI due to an issue installing OpenSSH. This _just works_ on Windows Server 2022. I have no clue what the root cause is. I think it might have something to do with Microsoft not publishing the files in the right location. Differential Revision: https://phab.mercurial-scm.org/D12630

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Sat, 04 Jun 2022 11:18:32 -0700
parents	5c8148cd7f13
children	3e84e001b6c1

rev	line source
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # aws.py - Automation code for Amazon Web Services
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 # no-check-code because Python 3 native.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10 import contextlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11 import copy
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12 import hashlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13 import json
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 import os
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 import pathlib
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	16 import subprocess
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 import time
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 import boto3
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	20 import botocore.exceptions
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	21
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	22 from .linux import BOOTSTRAP_DEBIAN
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	23 from .ssh import (
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	24 exec_command as ssh_exec_command,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	25 wait_for_ssh,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	26 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	27 from .winrm import (
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	28 run_powershell,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 wait_for_winrm,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	33 SOURCE_ROOT = pathlib.Path(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	34 os.path.abspath(__file__)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	35 ).parent.parent.parent.parent
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	37 INSTALL_WINDOWS_DEPENDENCIES = (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	38 SOURCE_ROOT / 'contrib' / 'install-windows-dependencies.ps1'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	39 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	40
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	41
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	42 INSTANCE_TYPES_WITH_STORAGE = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	43 'c5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	44 'd2',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	45 'h1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	46 'i3',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	47 'm5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	48 'm5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	49 'r5d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	50 'r5ad',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	51 'x1',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	52 'z1d',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	53 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	54
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	55
42647 8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	56 AMAZON_ACCOUNT_ID = '801119661308'
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	57 DEBIAN_ACCOUNT_ID = '379101102735'
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	58 DEBIAN_ACCOUNT_ID_2 = '136693071363'
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	59 UBUNTU_ACCOUNT_ID = '099720109477'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	60
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	61
49402 f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	62 WINDOWS_BASE_IMAGE_NAME = 'Windows_Server-2022-English-Full-Base-*'
42647 8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	63
8804aa6c07a0 automation: extract strings to constants Gregory Szorc <gregory.szorc@gmail.com> parents: 42646 diff changeset	64
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	65 KEY_PAIRS = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	66 'automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	69
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	70 SECURITY_GROUPS = {
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	71 'linux-dev-1': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	72 'description': 'Mercurial Linux instances that perform build/test automation',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	73 'ingress': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	74 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	75 'FromPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	76 'ToPort': 22,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	77 'IpProtocol': 'tcp',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	78 'IpRanges': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	79 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	80 'CidrIp': '0.0.0.0/0',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	81 'Description': 'SSH from entire Internet',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	82 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	83 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	84 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	85 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	86 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 'windows-dev-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 'description': 'Mercurial Windows instances that perform build automation',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 'ingress': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 'FromPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 'ToPort': 22,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97 'Description': 'SSH from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 'FromPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 'ToPort': 3389,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 'Description': 'RDP from entire Internet',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 'FromPort': 5985,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 'ToPort': 5986,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115 'IpProtocol': 'tcp',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 'IpRanges': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118 'CidrIp': '0.0.0.0/0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119 'Description': 'PowerShell Remoting (Windows Remote Management)',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121 ],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	122 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	126
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 IAM_ROLES = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 'ephemeral-ec2-role-1': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130 'description': 'Mercurial temporary EC2 instances',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131 'policy_arns': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 ASSUME_ROLE_POLICY_DOCUMENT = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	140 "Version": "2012-10-17",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141 "Statement": [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	142 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 "Effect": "Allow",
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 "Principal": {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145 "Service": "ec2.amazonaws.com"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147 "Action": "sts:AssumeRole"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	149 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151 '''.strip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	153
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	154 IAM_INSTANCE_PROFILES = {
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	155 'ephemeral-ec2-1': {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	156 'roles': [
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	157 'ephemeral-ec2-role-1',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	158 ],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	159 }
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	161
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	162
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	163 # User Data for Windows EC2 instance. Mainly used to set the password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164 # and configure WinRM.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	165 # Inspired by the User Data script used by Packer
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	166 # (from https://www.packer.io/intro/getting-started/build-image.html).
42064 0e9066db5e44 automation: use raw strings when there are backslashes Gregory Szorc <gregory.szorc@gmail.com> parents: 42024 diff changeset	167 WINDOWS_USER_DATA = r'''
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168 <powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170 # TODO enable this once we figure out what is failing.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171 #$ErrorActionPreference = "stop"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	173 # Set administrator password
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	174 net user Administrator "%s"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	175 wmic useraccount where "name='Administrator'" set PasswordExpires=FALSE
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176
49402 f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	177 # And set it via EC2Launch so it persists across reboots.
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	178 $config = & $env:ProgramFiles\Amazon\EC2Launch\EC2Launch.exe get-agent-config --format json \| ConvertFrom-Json
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	179 $config \| ConvertTo-Json -Depth 6 \| Out-File -encoding UTF8 $env:ProgramData/Amazon/EC2Launch/config/agent-config.yml
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	180 $setAdminAccount = @"
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	181 {
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	182 "task": "setAdminAccount",
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	183 "inputs": {
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	184 "password": {
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	185 "type": "static",
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	186 "data": "%s"
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	187 }
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	188 }
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	189 }
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	190 "@
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	191 $config.config \| %%{if($_.stage -eq 'preReady'){$_.tasks += (ConvertFrom-Json -InputObject $setAdminAccount)}}
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	192 $config \| ConvertTo-Json -Depth 6 \| Out-File -encoding UTF8 $env:ProgramData/Amazon/EC2Launch/config/agent-config.yml
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	193
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	194 # First, make sure WinRM can't be connected to
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	195 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new enable=yes action=block
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	196
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	197 # Delete any existing WinRM listeners
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	198 winrm delete winrm/config/listener?Address=*+Transport=HTTP 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	199 winrm delete winrm/config/listener?Address=*+Transport=HTTPS 2>$Null
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	200
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201 # Create a new WinRM listener and configure
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	202 winrm create winrm/config/listener?Address=*+Transport=HTTP
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	203 winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="0"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204 winrm set winrm/config '@{MaxTimeoutms="7200000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	205 winrm set winrm/config/service '@{AllowUnencrypted="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	206 winrm set winrm/config/service '@{MaxConcurrentOperationsPerUser="12000"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	207 winrm set winrm/config/service/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	208 winrm set winrm/config/client/auth '@{Basic="true"}'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	210 # Configure UAC to allow privilege elevation in remote shells
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 $Key = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212 $Setting = 'LocalAccountTokenFilterPolicy'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213 Set-ItemProperty -Path $Key -Name $Setting -Value 1 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214
43411 a77338d2bdab automation: avoid '~' in the temp directory on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 43234 diff changeset	215 # Avoid long usernames in the temp directory path because the '~' causes extra quoting in ssh output
a77338d2bdab automation: avoid '~' in the temp directory on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 43234 diff changeset	216 [System.Environment]::SetEnvironmentVariable('TMP', 'C:\Temp', [System.EnvironmentVariableTarget]::User)
a77338d2bdab automation: avoid '~' in the temp directory on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 43234 diff changeset	217 [System.Environment]::SetEnvironmentVariable('TEMP', 'C:\Temp', [System.EnvironmentVariableTarget]::User)
a77338d2bdab automation: avoid '~' in the temp directory on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 43234 diff changeset	218
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219 # Configure and restart the WinRM Service; Enable the required firewall exception
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220 Stop-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221 Set-Service -Name WinRM -StartupType Automatic
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222 netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new action=allow localip=any remoteip=any
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	223 Start-Service -Name WinRM
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	224
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	225 # Disable firewall on private network interfaces so prompts don't appear.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	226 Set-NetFirewallProfile -Name private -Enabled false
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	227 </powershell>
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	228 '''.lstrip()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	229
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	230
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	231 WINDOWS_BOOTSTRAP_POWERSHELL = '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	232 Write-Output "installing PowerShell dependencies"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233 Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235 Install-Module -Name OpenSSHUtils -RequiredVersion 0.0.2.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	237 Write-Output "installing OpenSSL server"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	238 Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	239 # Various tools will attempt to use older versions of .NET. So we enable
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	240 # the feature that provides them so it doesn't have to be auto-enabled
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	241 # later.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	242 Write-Output "enabling .NET Framework feature"
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	243 Install-WindowsFeature -Name Net-Framework-Core
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	244 '''
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	245
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	246
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	247 class AWSConnection:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	248 """Manages the state of a connection with AWS."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	249
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	250 def __init__(self, automation, region: str, ensure_ec2_state: bool = True):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	251 self.automation = automation
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	252 self.local_state_path = automation.state_path
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	253
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	254 self.prefix = 'hg-'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	255
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	256 self.session = boto3.session.Session(region_name=region)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257 self.ec2client = self.session.client('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 self.ec2resource = self.session.resource('ec2')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259 self.iamclient = self.session.client('iam')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	260 self.iamresource = self.session.resource('iam')
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	261 self.security_groups = {}
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	262
42304 dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	263 if ensure_ec2_state:
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	264 ensure_key_pairs(automation.state_path, self.ec2resource)
dd6a9723ae2b automation: don't create resources when deleting things Gregory Szorc <gregory.szorc@gmail.com> parents: 42303 diff changeset	265 self.security_groups = ensure_security_groups(self.ec2resource)
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	266 ensure_iam_state(self.iamclient, self.iamresource)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	267
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	268 def key_pair_path_private(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	269 """Path to a key pair private key file."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270 return self.local_state_path / 'keys' / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	271
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 def key_pair_path_public(self, name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 return self.local_state_path / 'keys' / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 def rsa_key_fingerprint(p: pathlib.Path):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 """Compute the fingerprint of an RSA private key."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279 # TODO use rsa package.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 res = subprocess.run(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	281 [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	282 'openssl',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	283 'pkcs8',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	284 '-in',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	285 str(p),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	286 '-nocrypt',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	287 '-topk8',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	288 '-outform',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	289 'DER',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	290 ],
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 capture_output=True,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	292 check=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	293 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 sha1 = hashlib.sha1(res.stdout).hexdigest()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 return ':'.join(a + b for a, b in zip(sha1[::2], sha1[1::2]))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299 def ensure_key_pairs(state_path: pathlib.Path, ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 remote_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 for kpi in ec2resource.key_pairs.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303 if kpi.name.startswith(prefix):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	304 remote_existing[kpi.name[len(prefix) :]] = kpi.key_fingerprint
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 # Validate that we have these keys locally.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307 key_path = state_path / 'keys'
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 key_path.mkdir(exist_ok=True, mode=0o700)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	309
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 def remove_remote(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311 print('deleting key pair %s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312 key = ec2resource.KeyPair(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313 key.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315 def remove_local(name):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	316 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319 print('removing %s' % pub_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 print('removing %s' % priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324 local_existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	325
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 for f in sorted(os.listdir(key_path)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327 if not f.startswith('keypair-') or not f.endswith('.pub'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	330 name = f[len('keypair-') : -len('.pub')]
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	332 pub_full = key_path / f
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	333 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	334
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	335 with open(pub_full, 'r', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	336 data = fh.read()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	337
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	338 if not data.startswith('ssh-rsa '):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	339 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	340 'unexpected format for key pair file: %s; removing' % pub_full
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	341 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342 pub_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 priv_full.unlink()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346 local_existing[name] = rsa_key_fingerprint(priv_full)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	347
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348 for name in sorted(set(remote_existing) \| set(local_existing)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	349 if name not in local_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	350 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	351 print('remote key %s does not exist locally' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	352 remove_remote(actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	353 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	354
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	355 elif name not in remote_existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	356 print('local key %s does not exist remotely' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360 elif remote_existing[name] != local_existing[name]:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	361 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	362 'key fingerprint mismatch for %s; '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	363 'removing from local and remote' % name
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	364 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365 remove_local(name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	366 remove_remote('%s%s' % (prefix, name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	367 del local_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	368 del remote_existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	369
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370 missing = KEY_PAIRS - set(remote_existing)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372 for name in sorted(missing):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	373 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	374 print('creating key pair %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	375
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376 priv_full = key_path / ('keypair-%s' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377 pub_full = key_path / ('keypair-%s.pub' % name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379 kp = ec2resource.create_key_pair(KeyName=actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381 with priv_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382 fh.write(kp.key_material)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383 fh.write('\n')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385 priv_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	386
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	387 # SSH public key can be extracted via `ssh-keygen`.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	388 with pub_full.open('w', encoding='ascii') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	389 subprocess.run(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	390 ['ssh-keygen', '-y', '-f', str(priv_full)],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	391 stdout=fh,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	392 check=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	393 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	394
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	395 pub_full.chmod(0o0600)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398 def delete_instance_profile(profile):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399 for role in profile.roles:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	400 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	401 'removing role %s from instance profile %s'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	402 % (role.name, profile.name)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	403 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404 profile.remove_role(RoleName=role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	405
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	406 print('deleting instance profile %s' % profile.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	407 profile.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	408
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	409
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	410 def ensure_iam_state(iamclient, iamresource, prefix='hg-'):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	411 """Ensure IAM state is in sync with our canonical definition."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	412
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	413 remote_profiles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	414
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	415 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	416 if profile.name.startswith(prefix):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	417 remote_profiles[profile.name[len(prefix) :]] = profile
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	418
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	419 for name in sorted(set(remote_profiles) - set(IAM_INSTANCE_PROFILES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	420 delete_instance_profile(remote_profiles[name])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	421 del remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	422
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	423 remote_roles = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	424
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	425 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	426 if role.name.startswith(prefix):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	427 remote_roles[role.name[len(prefix) :]] = role
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	428
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	429 for name in sorted(set(remote_roles) - set(IAM_ROLES)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430 role = remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434 del remote_roles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	435
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	436 # We've purged remote state that doesn't belong. Create missing
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	437 # instance profiles and roles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438 for name in sorted(set(IAM_INSTANCE_PROFILES) - set(remote_profiles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440 print('creating IAM instance profile %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442 profile = iamresource.create_instance_profile(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	443 InstanceProfileName=actual
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	444 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445 remote_profiles[name] = profile
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	446
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	447 waiter = iamclient.get_waiter('instance_profile_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	448 waiter.wait(InstanceProfileName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	449 print('IAM instance profile %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	450
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	451 for name in sorted(set(IAM_ROLES) - set(remote_roles)):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	452 entry = IAM_ROLES[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	453
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	454 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	455 print('creating IAM role %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	456
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	457 role = iamresource.create_role(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	458 RoleName=actual,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 Description=entry['description'],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	460 AssumeRolePolicyDocument=ASSUME_ROLE_POLICY_DOCUMENT,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	462
42305 8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	463 waiter = iamclient.get_waiter('role_exists')
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	464 waiter.wait(RoleName=actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	465 print('IAM role %s is available' % actual)
8dc22a209420 automation: wait for instance profiles and roles Gregory Szorc <gregory.szorc@gmail.com> parents: 42304 diff changeset	466
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	467 remote_roles[name] = role
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	468
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	469 for arn in entry['policy_arns']:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	470 print('attaching policy %s to %s' % (arn, role.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	471 role.attach_policy(PolicyArn=arn)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	472
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	473 # Now reconcile state of profiles.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	474 for name, meta in sorted(IAM_INSTANCE_PROFILES.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	475 profile = remote_profiles[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	476 wanted = {'%s%s' % (prefix, role) for role in meta['roles']}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	477 have = {role.name for role in profile.roles}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	478
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	479 for role in sorted(have - wanted):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	480 print('removing role %s from %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	481 profile.remove_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	482
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	483 for role in sorted(wanted - have):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	484 print('adding role %s to %s' % (role, profile.name))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	485 profile.add_role(RoleName=role)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	486
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	487
44734 828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	488 def find_image(ec2resource, owner_id, name, reverse_sort_field=None):
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	489 """Find an AMI by its owner ID and name."""
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	490
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	491 images = ec2resource.images.filter(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	492 Filters=[
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	493 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	494 'Name': 'owner-id',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	495 'Values': [owner_id],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	496 },
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	497 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	498 'Name': 'state',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	499 'Values': ['available'],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	500 },
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	501 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	502 'Name': 'image-type',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	503 'Values': ['machine'],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	504 },
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	505 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	506 'Name': 'name',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	507 'Values': [name],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	508 },
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	509 ]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	510 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511
44734 828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	512 if reverse_sort_field:
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	513 images = sorted(
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	514 images,
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	515 key=lambda image: getattr(image, reverse_sort_field),
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	516 reverse=True,
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	517 )
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	518
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	519 for image in images:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	520 return image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	521
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	522 raise Exception('unable to find image for %s' % name)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	523
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	524
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	525 def ensure_security_groups(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	526 """Ensure all necessary Mercurial security groups are present.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	527
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	528 All security groups are prefixed with ``hg-`` by default. Any security
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	529 groups having this prefix but aren't in our list are deleted.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	530 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	531 existing = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	532
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	533 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	534 if group.group_name.startswith(prefix):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	535 existing[group.group_name[len(prefix) :]] = group
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	536
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	537 purge = set(existing) - set(SECURITY_GROUPS)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	538
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	539 for name in sorted(purge):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	540 group = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	541 print('removing legacy security group: %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544 security_groups = {}
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546 for name, group in sorted(SECURITY_GROUPS.items()):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	547 if name in existing:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	548 security_groups[name] = existing[name]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551 actual = '%s%s' % (prefix, name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	552 print('adding security group %s' % actual)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	553
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	554 group_res = ec2resource.create_security_group(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	555 Description=group['description'],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	556 GroupName=actual,
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	557 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	558
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	559 group_res.authorize_ingress(
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	560 IpPermissions=group['ingress'],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	561 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	562
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	563 security_groups[name] = group_res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	564
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	565 return security_groups
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	566
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	568 def terminate_ec2_instances(ec2resource, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569 """Terminate all EC2 instances managed by us."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	570 waiting = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 for instance in ec2resource.instances.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573 if instance.state['Name'] == 'terminated':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	574 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	575
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	576 for tag in instance.tags or []:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	577 if tag['Key'] == 'Name' and tag['Value'].startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	578 print('terminating %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	580 waiting.append(instance)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	581
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	582 for instance in waiting:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	583 instance.wait_until_terminated()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	584
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	585
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586 def remove_resources(c, prefix='hg-'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	587 """Purge all of our resources in this EC2 region."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	588 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	589 iamresource = c.iamresource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	590
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	591 terminate_ec2_instances(ec2resource, prefix=prefix)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	592
42302 730edbd836d8 automation: only iterate over our AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 42064 diff changeset	593 for image in ec2resource.images.filter(Owners=['self']):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	594 if image.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595 remove_ami(ec2resource, image)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	596
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	597 for group in ec2resource.security_groups.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	598 if group.group_name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	599 print('removing security group %s' % group.group_name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	600 group.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	601
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	602 for profile in iamresource.instance_profiles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	603 if profile.name.startswith(prefix):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	604 delete_instance_profile(profile)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	605
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	606 for role in iamresource.roles.all():
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	607 if role.name.startswith(prefix):
42303 fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	608 for p in role.attached_policies.all():
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	609 print('detaching policy %s from %s' % (p.arn, role.name))
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	610 role.detach_policy(PolicyArn=p.arn)
fcb97cb91ff8 automation: detach policies before deleting role Gregory Szorc <gregory.szorc@gmail.com> parents: 42302 diff changeset	611
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	612 print('removing role %s' % role.name)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	613 role.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	614
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	615
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	616 def wait_for_ip_addresses(instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	617 """Wait for the public IP addresses of an iterable of instances."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	618 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	619 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	620 if not instance.public_ip_address:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	621 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	622 instance.reload()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	623 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	624
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	625 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	626 'public IP address for %s: %s'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	627 % (instance.id, instance.public_ip_address)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	628 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	629 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	630
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	631
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	632 def remove_ami(ec2resource, image):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	633 """Remove an AMI and its underlying snapshots."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	634 snapshots = []
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	635
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	636 for device in image.block_device_mappings:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	637 if 'Ebs' in device:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	638 snapshots.append(ec2resource.Snapshot(device['Ebs']['SnapshotId']))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	639
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	640 print('deregistering %s' % image.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	641 image.deregister()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	642
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	643 for snapshot in snapshots:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	644 print('deleting snapshot %s' % snapshot.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	645 snapshot.delete()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	646
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	647
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	648 def wait_for_ssm(ssmclient, instances):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	649 """Wait for SSM to come online for an iterable of instance IDs."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	650 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	651 res = ssmclient.describe_instance_information(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	652 Filters=[
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	653 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	654 'Key': 'InstanceIds',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	655 'Values': [i.id for i in instances],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	656 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	657 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	658 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	659
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	660 available = len(res['InstanceInformationList'])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	661 wanted = len(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	662
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	663 print('%d/%d instances available in SSM' % (available, wanted))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	664
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	665 if available == wanted:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	666 return
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	667
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	668 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	669
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	670
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	671 def run_ssm_command(ssmclient, instances, document_name, parameters):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	672 """Run a PowerShell script on an EC2 instance."""
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	673
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	674 res = ssmclient.send_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	675 InstanceIds=[i.id for i in instances],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	676 DocumentName=document_name,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	677 Parameters=parameters,
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	678 CloudWatchOutputConfig={
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	679 'CloudWatchOutputEnabled': True,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	680 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	681 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	682
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	683 command_id = res['Command']['CommandId']
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	684
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	685 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	686 while True:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	687 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	688 res = ssmclient.get_command_invocation(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	689 CommandId=command_id,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	690 InstanceId=instance.id,
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	691 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	692 except botocore.exceptions.ClientError as e:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	693 if e.response['Error']['Code'] == 'InvocationDoesNotExist':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	694 print('could not find SSM command invocation; waiting')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	695 time.sleep(1)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	696 continue
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	697 else:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	698 raise
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	699
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	700 if res['Status'] == 'Success':
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	701 break
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	702 elif res['Status'] in ('Pending', 'InProgress', 'Delayed'):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	703 time.sleep(2)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	704 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	705 raise Exception(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	706 'command failed on %s: %s' % (instance.id, res['Status'])
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	707 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	708
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	709
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	710 @contextlib.contextmanager
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	711 def temporary_ec2_instances(ec2resource, config):
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	712 """Create temporary EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	713
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	714 This is a proxy to ``ec2client.run_instances(**config)`` that takes care of
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	715 managing the lifecycle of the instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	716
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	717 When the context manager exits, the instances are terminated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	718
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	719 The context manager evaluates to the list of data structures
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	720 describing each created instance. The instances may not be available
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	721 for work immediately: it is up to the caller to wait for the instance
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	722 to start responding.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	724
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	725 ids = None
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	726
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	727 try:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	728 res = ec2resource.create_instances(**config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	729
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	730 ids = [i.id for i in res]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	731 print('started instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	732
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	733 yield res
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	734 finally:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	735 if ids:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	736 print('terminating instances: %s' % ' '.join(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	737 for instance in res:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	738 instance.terminate()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	739 print('terminated %d instances' % len(ids))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	740
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	741
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	742 @contextlib.contextmanager
43234 c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	743 def create_temp_windows_ec2_instances(
c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	744 c: AWSConnection, config, bootstrap: bool = False
c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	745 ):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	746 """Create temporary Windows EC2 instances.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	747
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	748 This is a higher-level wrapper around ``create_temp_ec2_instances()`` that
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	749 configures the Windows instance for Windows Remote Management. The emitted
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	750 instances will have a ``winrm_client`` attribute containing a
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	751 ``pypsrp.client.Client`` instance bound to the instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	752 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	753 if 'IamInstanceProfile' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	754 raise ValueError('IamInstanceProfile cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	755 if 'UserData' in config:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	756 raise ValueError('UserData cannot be provided in config')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	757
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	758 password = c.automation.default_password()
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	759
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	760 config = copy.deepcopy(config)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	761 config['IamInstanceProfile'] = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	762 'Name': 'hg-ephemeral-ec2-1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	763 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	764 config.setdefault('TagSpecifications', []).append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	765 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	766 'ResourceType': 'instance',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	767 'Tags': [{'Key': 'Name', 'Value': 'hg-temp-windows'}],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	768 }
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	769 )
43234 c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	770
c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	771 if bootstrap:
49402 f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	772 config['UserData'] = WINDOWS_USER_DATA % (password, password)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	773
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	774 with temporary_ec2_instances(c.ec2resource, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	775 wait_for_ip_addresses(instances)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	776
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	777 print('waiting for Windows Remote Management service...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	778
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	779 for instance in instances:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	780 client = wait_for_winrm(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	781 instance.public_ip_address, 'Administrator', password
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	782 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	783 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	784 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	785
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	786 yield instances
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	787
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	788
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	789 def resolve_fingerprint(fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	790 fingerprint = json.dumps(fingerprint, sort_keys=True)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	791 return hashlib.sha256(fingerprint.encode('utf-8')).hexdigest()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	792
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	793
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	794 def find_and_reconcile_image(ec2resource, name, fingerprint):
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	795 """Attempt to find an existing EC2 AMI with a name and fingerprint.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	796
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	797 If an image with the specified fingerprint is found, it is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	798 Otherwise None is returned.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	799
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	800 Existing images for the specified name that don't have the specified
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	801 fingerprint or are missing required metadata or deleted.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	802 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	803 # Find existing AMIs with this name and delete the ones that are invalid.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	804 # Store a reference to a good image so it can be returned one the
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	805 # image state is reconciled.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	806 images = ec2resource.images.filter(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	807 Filters=[{'Name': 'name', 'Values': [name]}]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	808 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	809
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	810 existing_image = None
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	811
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	812 for image in images:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	813 if image.tags is None:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	814 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	815 'image %s for %s lacks required tags; removing'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	816 % (image.id, image.name)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	817 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	818 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	819 else:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	820 tags = {t['Key']: t['Value'] for t in image.tags}
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	821
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	822 if tags.get('HGIMAGEFINGERPRINT') == fingerprint:
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	823 existing_image = image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	824 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	825 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	826 'image %s for %s has wrong fingerprint; removing'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	827 % (image.id, image.name)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	828 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	829 remove_ami(ec2resource, image)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	830
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	831 return existing_image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	832
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	833
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	834 def create_ami_from_instance(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	835 ec2client, instance, name, description, fingerprint
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	836 ):
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	837 """Create an AMI from a running instance.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	838
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	839 Returns the ``ec2resource.Image`` representing the created AMI.
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	840 """
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	841 instance.stop()
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	842
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	843 ec2client.get_waiter('instance_stopped').wait(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	844 InstanceIds=[instance.id],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	845 WaiterConfig={
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	846 'Delay': 5,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	847 },
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	848 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	849 print('%s is stopped' % instance.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	850
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	851 image = instance.create_image(
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	852 Name=name,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	853 Description=description,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	854 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	855
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	856 image.create_tags(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	857 Tags=[
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	858 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	859 'Key': 'HGIMAGEFINGERPRINT',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	860 'Value': fingerprint,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	861 },
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	862 ]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	863 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	864
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	865 print('waiting for image %s' % image.id)
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	866
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	867 ec2client.get_waiter('image_available').wait(
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	868 ImageIds=[image.id],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	869 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	870
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	871 print('image %s available as %s' % (image.id, image.name))
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	872
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	873 return image
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	874
195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	875
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	876 def ensure_linux_dev_ami(c: AWSConnection, distro='debian10', prefix='hg-'):
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	877 """Ensures a Linux development AMI is available and up-to-date.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	878
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	879 Returns an ``ec2.Image`` of either an existing AMI or a newly-built one.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	880 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	881 ec2client = c.ec2client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	882 ec2resource = c.ec2resource
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	883
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	884 name = '%s%s-%s' % (prefix, 'linux-dev', distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	885
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	886 if distro == 'debian9':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	887 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	888 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	889 DEBIAN_ACCOUNT_ID,
43019 6952d42f9158 automation: use latest AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 43018 diff changeset	890 'debian-stretch-hvm-x86_64-gp2-2019-09-08-17994',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	891 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	892 ssh_username = 'admin'
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	893 elif distro == 'debian10':
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	894 image = find_image(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	895 ec2resource,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	896 DEBIAN_ACCOUNT_ID_2,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	897 'debian-10-amd64-20190909-10',
43020 d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	898 )
d1d919f679f7 automation: support and use Debian Buster by default Gregory Szorc <gregory.szorc@gmail.com> parents: 43019 diff changeset	899 ssh_username = 'admin'
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	900 elif distro == 'ubuntu18.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	901 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	902 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	903 UBUNTU_ACCOUNT_ID,
43019 6952d42f9158 automation: use latest AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 43018 diff changeset	904 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190918',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	905 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	906 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	907 elif distro == 'ubuntu19.04':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	908 image = find_image(
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	909 ec2resource,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	910 UBUNTU_ACCOUNT_ID,
43019 6952d42f9158 automation: use latest AMIs Gregory Szorc <gregory.szorc@gmail.com> parents: 43018 diff changeset	911 'ubuntu/images/hvm-ssd/ubuntu-disco-19.04-amd64-server-20190918',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	912 )
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	913 ssh_username = 'ubuntu'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	914 else:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	915 raise ValueError('unsupported Linux distro: %s' % distro)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	916
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	917 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	918 'BlockDeviceMappings': [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	919 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	920 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	921 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	922 'DeleteOnTermination': True,
43018 92c8bae84e7a automation: increase size of Linux AMI build volume Gregory Szorc <gregory.szorc@gmail.com> parents: 43015 diff changeset	923 'VolumeSize': 10,
48377 2d6940811067 automation: use gp3 volume type Gregory Szorc <gregory.szorc@gmail.com> parents: 48376 diff changeset	924 'VolumeType': 'gp3',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	925 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	926 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	927 ],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	928 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	929 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	930 'InstanceInitiatedShutdownBehavior': 'stop',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	931 # 8 VCPUs for compiling Python.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	932 'InstanceType': 't3.2xlarge',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	933 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	934 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	935 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	936 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	937 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	938
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	939 requirements3_path = (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	940 pathlib.Path(__file__).parent.parent / 'linux-requirements-py3.txt'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	941 )
47206 546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	942 requirements35_path = (
546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	943 pathlib.Path(__file__).parent.parent / 'linux-requirements-py3.5.txt'
546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	944 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	945 with requirements3_path.open('r', encoding='utf-8') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	946 requirements3 = fh.read()
47206 546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	947 with requirements35_path.open('r', encoding='utf-8') as fh:
546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	948 requirements35 = fh.read()
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	949
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	950 # Compute a deterministic fingerprint to determine whether image needs to
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	951 # be regenerated.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	952 fingerprint = resolve_fingerprint(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	953 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	954 'instance_config': config,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	955 'bootstrap_script': BOOTSTRAP_DEBIAN,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	956 'requirements_py3': requirements3,
47206 546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	957 'requirements_py35': requirements35,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	958 }
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	959 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	960
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	961 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	962
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	963 if existing_image:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	964 return existing_image
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	965
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	966 print('no suitable %s image found; creating one...' % name)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	967
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	968 with temporary_ec2_instances(ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	969 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	970
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	971 instance = instances[0]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	972
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	973 client = wait_for_ssh(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	974 instance.public_ip_address,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	975 22,
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	976 username=ssh_username,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	977 key_filename=str(c.key_pair_path_private('automation')),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	978 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	979
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	980 home = '/home/%s' % ssh_username
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	981
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	982 with client:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	983 print('connecting to SSH server')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	984 sftp = client.open_sftp()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	985
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	986 print('uploading bootstrap files')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	987 with sftp.open('%s/bootstrap' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	988 fh.write(BOOTSTRAP_DEBIAN)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	989 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	990
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	991 with sftp.open('%s/requirements-py3.txt' % home, 'wb') as fh:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	992 fh.write(requirements3)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	993 fh.chmod(0o0700)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	994
47206 546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	995 with sftp.open('%s/requirements-py3.5.txt' % home, 'wb') as fh:
546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	996 fh.write(requirements35)
546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	997 fh.chmod(0o0700)
546e812a1c2d automation: create Python 3.5 variant of requirements.txt Gregory Szorc <gregory.szorc@gmail.com> parents: 45957 diff changeset	998
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	999 print('executing bootstrap')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1000 chan, stdin, stdout = ssh_exec_command(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1001 client, '%s/bootstrap' % home
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1002 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1003 stdin.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1004
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1005 for line in stdout:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1006 print(line, end='')
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1007
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1008 res = chan.recv_exit_status()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1009 if res:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1010 raise Exception('non-0 exit from bootstrap: %d' % res)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1011
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1012 print(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1013 'bootstrap completed; stopping %s to create %s'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1014 % (instance.id, name)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1015 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1016
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1017 return create_ami_from_instance(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1018 ec2client,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1019 instance,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1020 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1021 'Mercurial Linux development environment',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1022 fingerprint,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1023 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1024
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1025
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1026 @contextlib.contextmanager
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1027 def temporary_linux_dev_instances(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1028 c: AWSConnection,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1029 image,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1030 instance_type,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1031 prefix='hg-',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1032 ensure_extra_volume=False,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1033 ):
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1034 """Create temporary Linux development EC2 instances.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1035
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1036 Context manager resolves to a list of ``ec2.Instance`` that were created
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1037 and are running.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1038
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1039 ``ensure_extra_volume`` can be set to ``True`` to require that instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1040 have a 2nd storage volume available other than the primary AMI volume.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1041 For instance types with instance storage, this does nothing special.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1042 But for instance types without instance storage, an additional EBS volume
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1043 will be added to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1044
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1045 Instances have an ``ssh_client`` attribute containing a paramiko SSHClient
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1046 instance bound to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1047
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1048 Instances have an ``ssh_private_key_path`` attributing containing the
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1049 str path to the SSH private key to connect to the instance.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1050 """
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1051
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1052 block_device_mappings = [
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1053 {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1054 'DeviceName': image.block_device_mappings[0]['DeviceName'],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1055 'Ebs': {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1056 'DeleteOnTermination': True,
42719 3e3fb15bfeea automation: increase root volume size on Linux Gregory Szorc <gregory.szorc@gmail.com> parents: 42648 diff changeset	1057 'VolumeSize': 12,
48377 2d6940811067 automation: use gp3 volume type Gregory Szorc <gregory.szorc@gmail.com> parents: 48376 diff changeset	1058 'VolumeType': 'gp3',
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1059 },
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1060 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1061 ]
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1062
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1063 # This is not an exhaustive list of instance types having instance storage.
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1064 # But
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1065 if ensure_extra_volume and not instance_type.startswith(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1066 tuple(INSTANCE_TYPES_WITH_STORAGE)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1067 ):
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1068 main_device = block_device_mappings[0]['DeviceName']
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1069
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1070 if main_device == 'xvda':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1071 second_device = 'xvdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1072 elif main_device == '/dev/sda1':
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1073 second_device = '/dev/sdb'
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1074 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1075 raise ValueError(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1076 'unhandled primary EBS device name: %s' % main_device
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1077 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1078
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1079 block_device_mappings.append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1080 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1081 'DeviceName': second_device,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1082 'Ebs': {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1083 'DeleteOnTermination': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1084 'VolumeSize': 8,
48377 2d6940811067 automation: use gp3 volume type Gregory Szorc <gregory.szorc@gmail.com> parents: 48376 diff changeset	1085 'VolumeType': 'gp3',
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1086 },
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1087 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1088 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1089
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1090 config = {
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1091 'BlockDeviceMappings': block_device_mappings,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1092 'EbsOptimized': True,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1093 'ImageId': image.id,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1094 'InstanceInitiatedShutdownBehavior': 'terminate',
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1095 'InstanceType': instance_type,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1096 'KeyName': '%sautomation' % prefix,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1097 'MaxCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1098 'MinCount': 1,
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1099 'SecurityGroupIds': [c.security_groups['linux-dev-1'].id],
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1100 }
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1101
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1102 with temporary_ec2_instances(c.ec2resource, config) as instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1103 wait_for_ip_addresses(instances)
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1104
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1105 ssh_private_key_path = str(c.key_pair_path_private('automation'))
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1106
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1107 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1108 client = wait_for_ssh(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1109 instance.public_ip_address,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1110 22,
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1111 username='hg',
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1112 key_filename=ssh_private_key_path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1113 )
42312 65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1114
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1115 instance.ssh_client = client
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1116 instance.ssh_private_key_path = ssh_private_key_path
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1117
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1118 try:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1119 yield instances
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1120 finally:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1121 for instance in instances:
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1122 instance.ssh_client.close()
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1123
65b3ef162b39 automation: initial support for running Linux tests Gregory Szorc <gregory.szorc@gmail.com> parents: 42311 diff changeset	1124
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1125 def ensure_windows_dev_ami(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1126 c: AWSConnection,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1127 prefix='hg-',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1128 base_image_name=WINDOWS_BASE_IMAGE_NAME,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1129 ):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1130 """Ensure Windows Development AMI is available and up-to-date.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1131
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1132 If necessary, a modern AMI will be built by starting a temporary EC2
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1133 instance and bootstrapping it.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1134
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1135 Obsolete AMIs will be deleted so there is only a single AMI having the
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1136 desired name.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1137
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1138 Returns an ``ec2.Image`` of either an existing AMI or a newly-built
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1139 one.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1140 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1141 ec2client = c.ec2client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1142 ec2resource = c.ec2resource
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1143 ssmclient = c.session.client('ssm')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1144
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1145 name = '%s%s' % (prefix, 'windows-dev')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1146
44734 828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	1147 image = find_image(
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	1148 ec2resource,
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	1149 AMAZON_ACCOUNT_ID,
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	1150 base_image_name,
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	1151 reverse_sort_field="name",
828d3277618a automation: always use latest Windows AMI Gregory Szorc <gregory.szorc@gmail.com> parents: 43841 diff changeset	1152 )
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1153
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1154 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1155 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1156 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1157 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1158 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1159 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1160 'VolumeSize': 32,
48377 2d6940811067 automation: use gp3 volume type Gregory Szorc <gregory.szorc@gmail.com> parents: 48376 diff changeset	1161 'VolumeType': 'gp3',
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1162 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1163 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1164 ],
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1165 'ImageId': image.id,
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1166 'InstanceInitiatedShutdownBehavior': 'stop',
48376 ae28d37f5969 automation: use m6i instances Gregory Szorc <gregory.szorc@gmail.com> parents: 47206 diff changeset	1167 'InstanceType': 'm6i.large',
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1168 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1169 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1170 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1171 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1172 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1173
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1174 commands = [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1175 # Need to start the service so sshd_config is generated.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1176 'Start-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1177 'Write-Output "modifying sshd_config"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1178 r'$content = Get-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1179 '$content = $content -replace "Match Group administrators","" -replace "AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys",""',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1180 r'$content \| Set-Content C:\ProgramData\ssh\sshd_config',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1181 'Import-Module OpenSSHUtils',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1182 r'Repair-SshdConfigPermission C:\ProgramData\ssh\sshd_config -Confirm:$false',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1183 'Restart-Service sshd',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1184 'Write-Output "installing OpenSSL client"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1185 'Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1186 'Set-Service -Name sshd -StartupType "Automatic"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1187 'Write-Output "OpenSSH server running"',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1188 ]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1189
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1190 with INSTALL_WINDOWS_DEPENDENCIES.open('r', encoding='utf-8') as fh:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1191 commands.extend(l.rstrip() for l in fh)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1192
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1193 # Disable Windows Defender when bootstrapping because it just slows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1194 # things down.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1195 commands.insert(0, 'Set-MpPreference -DisableRealtimeMonitoring $true')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1196 commands.append('Set-MpPreference -DisableRealtimeMonitoring $false')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1197
49402 f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	1198 # Trigger shutdown to prepare for imaging.
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	1199 commands.append(
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	1200 'Stop-Computer -ComputerName localhost',
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	1201 )
f1dcddb7f328 automation: transition to Windows Server 2022 Gregory Szorc <gregory.szorc@gmail.com> parents: 48936 diff changeset	1202
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1203 # Compute a deterministic fingerprint to determine whether image needs
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1204 # to be regenerated.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1205 fingerprint = resolve_fingerprint(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1206 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1207 'instance_config': config,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1208 'user_data': WINDOWS_USER_DATA,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1209 'initial_bootstrap': WINDOWS_BOOTSTRAP_POWERSHELL,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1210 'bootstrap_commands': commands,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1211 'base_image_name': base_image_name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1212 }
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1213 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1214
42311 195dcc10b3d7 automation: move image operations to own functions Gregory Szorc <gregory.szorc@gmail.com> parents: 42307 diff changeset	1215 existing_image = find_and_reconcile_image(ec2resource, name, fingerprint)
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1216
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1217 if existing_image:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1218 return existing_image
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1219
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1220 print('no suitable Windows development image found; creating one...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1221
43234 c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	1222 with create_temp_windows_ec2_instances(
c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	1223 c, config, bootstrap=True
c09e8ac3f61f automation: schedule an EC2Launch run on next boot Gregory Szorc <gregory.szorc@gmail.com> parents: 43076 diff changeset	1224 ) as instances:
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1225 assert len(instances) == 1
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1226 instance = instances[0]
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1227
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1228 wait_for_ssm(ssmclient, [instance])
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1229
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1230 # On first boot, install various Windows updates.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1231 # We would ideally use PowerShell Remoting for this. However, there are
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1232 # trust issues that make it difficult to invoke Windows Update
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1233 # remotely. So we use SSM, which has a mechanism for running Windows
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1234 # Update.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1235 print('installing Windows features...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1236 run_ssm_command(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1237 ssmclient,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1238 [instance],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1239 'AWS-RunPowerShellScript',
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1240 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1241 'commands': WINDOWS_BOOTSTRAP_POWERSHELL.split('\n'),
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1242 },
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1243 )
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1244
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1245 # Reboot so all updates are fully applied.
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1246 #
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1247 # We don't use instance.reboot() here because it is asynchronous and
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1248 # we don't know when exactly the instance has rebooted. It could take
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1249 # a while to stop and we may start trying to interact with the instance
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1250 # before it has rebooted.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1251 print('rebooting instance %s' % instance.id)
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1252 instance.stop()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1253 ec2client.get_waiter('instance_stopped').wait(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1254 InstanceIds=[instance.id],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1255 WaiterConfig={
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1256 'Delay': 5,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44734 diff changeset	1257 },
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1258 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1259
42307 e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1260 instance.start()
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1261 wait_for_ip_addresses([instance])
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1262
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1263 # There is a race condition here between the User Data PS script running
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1264 # and us connecting to WinRM. This can manifest as
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1265 # "AuthorizationManager check failed" failures during run_powershell().
e570106beda1 automation: shore up rebooting behavior Gregory Szorc <gregory.szorc@gmail.com> parents: 42305 diff changeset	1266 # TODO figure out a workaround.
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1267
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1268 print('waiting for Windows Remote Management to come back...')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1269 client = wait_for_winrm(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1270 instance.public_ip_address,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1271 'Administrator',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1272 c.automation.default_password(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1273 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1274 print('established WinRM connection to %s' % instance.id)
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1275 instance.winrm_client = client
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1276
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1277 print('bootstrapping instance...')
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1278 run_powershell(instance.winrm_client, '\n'.join(commands))
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1279
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1280 print('bootstrap completed; stopping %s to create image' % instance.id)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1281 return create_ami_from_instance(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1282 ec2client,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1283 instance,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1284 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1285 'Mercurial Windows development environment',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1286 fingerprint,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1287 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1288
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1289
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1290 @contextlib.contextmanager
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1291 def temporary_windows_dev_instances(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1292 c: AWSConnection,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1293 image,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1294 instance_type,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1295 prefix='hg-',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1296 disable_antivirus=False,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1297 ):
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1298 """Create a temporary Windows development EC2 instance.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1299
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1300 Context manager resolves to the list of ``EC2.Instance`` that were created.
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1301 """
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1302 config = {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1303 'BlockDeviceMappings': [
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1304 {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1305 'DeviceName': '/dev/sda1',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1306 'Ebs': {
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1307 'DeleteOnTermination': True,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1308 'VolumeSize': 32,
48377 2d6940811067 automation: use gp3 volume type Gregory Szorc <gregory.szorc@gmail.com> parents: 48376 diff changeset	1309 'VolumeType': 'gp3',
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1310 },
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1311 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1312 ],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1313 'ImageId': image.id,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1314 'InstanceInitiatedShutdownBehavior': 'stop',
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1315 'InstanceType': instance_type,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1316 'KeyName': '%sautomation' % prefix,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1317 'MaxCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1318 'MinCount': 1,
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1319 'SecurityGroupIds': [c.security_groups['windows-dev-1'].id],
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1320 }
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1321
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1322 with create_temp_windows_ec2_instances(c, config) as instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1323 if disable_antivirus:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1324 for instance in instances:
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1325 run_powershell(
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1326 instance.winrm_client,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1327 'Set-MpPreference -DisableRealtimeMonitoring $true',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 43020 diff changeset	1328 )
42024 b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1329
b05a3e28cf24 automation: perform tasks on remote machines Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1330 yield instances

Mercurial > public > mercurial-scm > hg-stable

annotate contrib/automation/hgautomation/aws.py @ 49402:f1dcddb7f328 stable