CircleCIとterraformとGitHubのプライベートリポジトリで3環境を構成する
表題の件、結局以下のようにしてみた。当初は単一のアクセスキー/シークレットキーを使ってクロスアカウントのAssumeRoleで切り分けようかと思ったものの、AssumeRoleの切り替えに一時セッショントークンの取得などが必要で余計にややこしくなることから、システムユーザについてはAWSアカウント別に相当するユーザを作ってしまうことにした(人間のユーザはAssumeRoleで運用している)。
prodは本番用AWSアカウント、stgとdevは開発用AWSアカウント。各環境は別VPC。それぞれPowerUserポリシーのterraformerというIAMユーザを定義済み。
通常CircleCIでAWSと連携する場合は「AWS Permissions」にアクセスキーとシークレットキーを書くが、今回はそこには何も書かずに「Environment Variables」に「DEV_AWS_ACCESS_KEY_ID」「DEV_AWS_SECRET_ACCESS_KEY」のような感じでdev/stg/prodの分のアクセスキーを記述。
S3バケットとして対応するAWSアカウントに「company01-dev-terraform」「company01-stg-terraform」「company01-prod-terraform」というのを用意してそこで各環境のtfstateを管理(初回はバケットだけ作っておけば実際に何も入ってない状態でもremote pullとかは正しく動作する)。
git-flowとはちょっと違う運用で、developにpushすると開発環境のplan、release/developmentにpush(developをマージ)すると開発環境へのapply、release/stagingにpush(developをマージ)するとステージング環境へのapply、masterにpush(developをマージ)すると本番環境のplan、release/productionにpushする(masterをマージ)と本番環境へのapply。
GitHub上はOrganizationでプライベートリポジトリを作って運用しているが、当該リポジトリの[Settings]-[Branches]-[Protected branches]から、「master」「release/production」へのpush権限を別途作成したチーム「Administrators」に絞っている。
ちなみにアクセスキーを環境変数で与えている割に「provider "aws"」に「profile」を設定しているが、これは同じリソースを使って人間が手動で流すときに自身の「~/.aws/config」の設定で切り替えられるようにしたもの。CircleCI自体では使っていない。
- ci/
- terraform-build.sh
- terraform-install.sh
- terraform-validate.sh
- env/
- dev/
- main.tf
- prod/
- main.tf
- stg/
- main.tf
- dev/
- module/
- vpc/
- main.tf
- outputs.tf
- variables.tf
- web_cluster/
- main.tf
- outputs.tf
- variables.tf
- ...
- vpc/
- .gitignore
- circle.yml
- README.md
ci/terraform-build.sh
#!/bin/bash set -xe TF_ENV_DIR=$1 TF_BUCKET=$2 ROOT_DIR=$(git rev-parse --show-toplevel) cd "${ROOT_DIR}/${TF_ENV_DIR}" terraform get -update terraform remote config -backend=s3 -backend-config="bucket=${TF_BUCKET}" -backend-config="key=common-infra/terraform.tfstate" -backend-config="region=ap-northeast-1" terraform apply terraform remote push
ci/terraform-install.sh
#!/bin/bash set -xe cd ~/bin TF_VERSION=0.6.16 if [ ! -e ~/bin/terraform ]; then wget https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip unzip terraform_${TF_VERSION}_linux_amd64.zip rm terraform_${TF_VERSION}_linux_amd64.zip fi
ci/terraform-validate.sh
#!/bin/bash set -xe TF_ENV_DIR=$1 TF_BUCKET=$2 ROOT_DIR=$(git rev-parse --show-toplevel) cd "${ROOT_DIR}/${TF_ENV_DIR}" terraform get -update terraform remote config -backend=s3 -backend-config="bucket=${TF_BUCKET}" -backend-config="key=common-infra/terraform.tfstate" -backend-config="region=ap-northeast-1" terraform plan
env/dev/main.tf
provider "aws" { profile = "closed-aws-account" region = "ap-northeast-1" } module "vpc_dev" { source = "../../module/vpc" env = "dev" service = "common" cidr = "10.x.x.x/x" public_subnets = "10.x.x.x/x,10.x.x.x/x" private_subnets = "10.x.x.x/x,10.x.x.x/x" azs = "ap-northeast-1a,ap-northeast-1c" } module "service01_dev" { source = "../../module/web_cluster" env = "dev" service = "service01" ... } module "service02_dev" { source = "../../module/web_cluster" env = "dev" service = "service02" ... }
env/prod/main.tf
provider "aws" { profile = "open-aws-account" region = "ap-northeast-1" } module "vpc_prod" { source = "../../module/vpc" env = "prod" service = "common" cidr = "10.x.x.x/x" public_subnets = "10.x.x.x/x,10.x.x.x/x" private_subnets = "10.x.x.x/x,10.x.x.x/x" azs = "ap-northeast-1a,ap-northeast-1c" } module "service01_prod" { source = "../../module/web_cluster" env = "prod" service = "service01" ... } module "service02_prod" { source = "../../module/web_cluster" env = "prod" service = "service02" ... }
env/stg/main.tf
provider "aws" { profile = "closed-aws-account" region = "ap-northeast-1" } module "vpc_stg" { source = "../../module/vpc" env = "stg" service = "common" cidr = "10.x.x.x/x" public_subnets = "10.x.x.x/x,10.x.x.x/x" private_subnets = "10.x.x.x/x,10.x.x.x/x" azs = "ap-northeast-1a,ap-northeast-1c" } module "service01_stg" { source = "../../module/web_cluster" env = "stg" service = "service01" ... } module "service02_stg" { source = "../../module/web_cluster" env = "stg" service = "service02" ... }
.gitignore
.terraform *.tfstate *.tfstate.backup
circle.yml
general: branches: only: - release/development - release/staging - release/production - develop - master machine: environment: PATH: "${HOME}/bin:${PATH}" dependencies: cache_directories: - "~/bin" pre: - | mkdir -p ~/bin bash ./ci/terraform-install.sh test: override: - | if [ "${CIRCLE_BRANCH}" = "release/development" ] || [ "${CIRCLE_BRANCH}" = "develop" ]; then AWS_ACCESS_KEY_ID=${DEV_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${DEV_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-validate.sh env/dev company01-dev-terraform elif [ "${CIRCLE_BRANCH}" = "release/staging" ]; then AWS_ACCESS_KEY_ID=${STG_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${STG_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-validate.sh env/stg company01-stg-terraform elif [ "${CIRCLE_BRANCH}" = "release/production" ] || [ "${CIRCLE_BRANCH}" = "master" ]; then AWS_ACCESS_KEY_ID=${PROD_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${PROD_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-validate.sh env/prod company01-prod-terraform fi deployment: development: branch: release/development commands: - AWS_ACCESS_KEY_ID=${DEV_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${DEV_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-build.sh env/dev company01-dev-terraform staging: branch: release/staging commands: - AWS_ACCESS_KEY_ID=${STG_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${STG_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-build.sh env/stg company01-stg-terraform production: branch: release/production commands: - AWS_ACCESS_KEY_ID=${PROD_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${PROD_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-build.sh env/prod company01-prod-terraform