CircleCIとterraformとGitHubのプライベートリポジトリで3環境を構成する


表題の件、結局以下のようにしてみた。当初は単一のアクセスキー/シークレットキーを使ってクロスアカウントのAssumeRoleで切り分けようかと思ったものの、AssumeRoleの切り替えに一時セッショントークンの取得などが必要で余計にややこしくなることから、システムユーザについてはAWSアカウント別に相当するユーザを作ってしまうことにした(人間のユーザはAssumeRoleで運用している)。


prodは本番用AWSアカウント、stgとdevは開発用AWSアカウント。各環境は別VPC。それぞれPowerUserポリシーのterraformerというIAMユーザを定義済み。
通常CircleCIでAWSと連携する場合は「AWS Permissions」にアクセスキーとシークレットキーを書くが、今回はそこには何も書かずに「Environment Variables」に「DEV_AWS_ACCESS_KEY_ID」「DEV_AWS_SECRET_ACCESS_KEY」のような感じでdev/stg/prodの分のアクセスキーを記述。


S3バケットとして対応するAWSアカウントに「company01-dev-terraform」「company01-stg-terraform」「company01-prod-terraform」というのを用意してそこで各環境のtfstateを管理(初回はバケットだけ作っておけば実際に何も入ってない状態でもremote pullとかは正しく動作する)。


git-flowとはちょっと違う運用で、developにpushすると開発環境のplan、release/developmentにpush(developをマージ)すると開発環境へのapply、release/stagingにpush(developをマージ)するとステージング環境へのapply、masterにpush(developをマージ)すると本番環境のplan、release/productionにpushする(masterをマージ)と本番環境へのapply。


GitHub上はOrganizationでプライベートリポジトリを作って運用しているが、当該リポジトリの[Settings]-[Branches]-[Protected branches]から、「master」「release/production」へのpush権限を別途作成したチーム「Administrators」に絞っている。


ちなみにアクセスキーを環境変数で与えている割に「provider "aws"」に「profile」を設定しているが、これは同じリソースを使って人間が手動で流すときに自身の「~/.aws/config」の設定で切り替えられるようにしたもの。CircleCI自体では使っていない。

  • ci/
    • terraform-build.sh
    • terraform-install.sh
    • terraform-validate.sh
  • env/
    • dev/
      • main.tf
    • prod/
      • main.tf
    • stg/
      • main.tf
  • module/
    • vpc/
      • main.tf
      • outputs.tf
      • variables.tf
    • web_cluster/
      • main.tf
      • outputs.tf
      • variables.tf
    • ...
  • .gitignore
  • circle.yml
  • README.md

ci/terraform-build.sh

#!/bin/bash

set -xe

TF_ENV_DIR=$1
TF_BUCKET=$2

ROOT_DIR=$(git rev-parse --show-toplevel)

cd "${ROOT_DIR}/${TF_ENV_DIR}"

terraform get -update
terraform remote config -backend=s3 -backend-config="bucket=${TF_BUCKET}" -backend-config="key=common-infra/terraform.tfstate" -backend-config="region=ap-northeast-1"
terraform apply
terraform remote push

ci/terraform-install.sh

#!/bin/bash

set -xe

cd ~/bin

TF_VERSION=0.6.16

if [ ! -e ~/bin/terraform ]; then
  wget https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip
  unzip terraform_${TF_VERSION}_linux_amd64.zip
  rm terraform_${TF_VERSION}_linux_amd64.zip
fi

ci/terraform-validate.sh

#!/bin/bash

set -xe

TF_ENV_DIR=$1
TF_BUCKET=$2

ROOT_DIR=$(git rev-parse --show-toplevel)

cd "${ROOT_DIR}/${TF_ENV_DIR}"

terraform get -update
terraform remote config -backend=s3 -backend-config="bucket=${TF_BUCKET}" -backend-config="key=common-infra/terraform.tfstate" -backend-config="region=ap-northeast-1"
terraform plan

env/dev/main.tf

provider "aws" {
  profile = "closed-aws-account"
  region = "ap-northeast-1"
}

module "vpc_dev" {
  source = "../../module/vpc"
  env = "dev"
  service = "common"
  cidr = "10.x.x.x/x"
  public_subnets = "10.x.x.x/x,10.x.x.x/x"
  private_subnets = "10.x.x.x/x,10.x.x.x/x"
  azs = "ap-northeast-1a,ap-northeast-1c"
}

module "service01_dev" {
  source = "../../module/web_cluster"
  env = "dev"
  service = "service01"
  ...
}

module "service02_dev" {
  source = "../../module/web_cluster"
  env = "dev"
  service = "service02"
  ...
}

env/prod/main.tf

provider "aws" {
  profile = "open-aws-account"
  region = "ap-northeast-1"
}

module "vpc_prod" {
  source = "../../module/vpc"
  env = "prod"
  service = "common"
  cidr = "10.x.x.x/x"
  public_subnets = "10.x.x.x/x,10.x.x.x/x"
  private_subnets = "10.x.x.x/x,10.x.x.x/x"
  azs = "ap-northeast-1a,ap-northeast-1c"
}

module "service01_prod" {
  source = "../../module/web_cluster"
  env = "prod"
  service = "service01"
  ...
}

module "service02_prod" {
  source = "../../module/web_cluster"
  env = "prod"
  service = "service02"
  ...
}

env/stg/main.tf

provider "aws" {
  profile = "closed-aws-account"
  region = "ap-northeast-1"
}

module "vpc_stg" {
  source = "../../module/vpc"
  env = "stg"
  service = "common"
  cidr = "10.x.x.x/x"
  public_subnets = "10.x.x.x/x,10.x.x.x/x"
  private_subnets = "10.x.x.x/x,10.x.x.x/x"
  azs = "ap-northeast-1a,ap-northeast-1c"
}

module "service01_stg" {
  source = "../../module/web_cluster"
  env = "stg"
  service = "service01"
  ...
}

module "service02_stg" {
  source = "../../module/web_cluster"
  env = "stg"
  service = "service02"
  ...
}

.gitignore

.terraform
*.tfstate
*.tfstate.backup

circle.yml

general:
  branches:
    only:
      - release/development
      - release/staging
      - release/production
      - develop
      - master

machine:
  environment:
    PATH: "${HOME}/bin:${PATH}"

dependencies:
  cache_directories:
    - "~/bin"
  pre:
    - |
      mkdir -p ~/bin
      bash ./ci/terraform-install.sh

test:
  override:
    - |
      if [ "${CIRCLE_BRANCH}" = "release/development" ] || [ "${CIRCLE_BRANCH}" = "develop" ]; then
        AWS_ACCESS_KEY_ID=${DEV_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${DEV_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-validate.sh env/dev company01-dev-terraform
      elif [ "${CIRCLE_BRANCH}" = "release/staging" ]; then
        AWS_ACCESS_KEY_ID=${STG_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${STG_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-validate.sh env/stg company01-stg-terraform
      elif [ "${CIRCLE_BRANCH}" = "release/production" ] || [ "${CIRCLE_BRANCH}" = "master" ]; then
        AWS_ACCESS_KEY_ID=${PROD_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${PROD_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-validate.sh env/prod company01-prod-terraform
      fi

deployment:
  development:
    branch: release/development
    commands:
      - AWS_ACCESS_KEY_ID=${DEV_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${DEV_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-build.sh env/dev company01-dev-terraform
  staging:
    branch: release/staging
    commands:
      - AWS_ACCESS_KEY_ID=${STG_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${STG_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-build.sh env/stg company01-stg-terraform
  production:
    branch: release/production
    commands:
      - AWS_ACCESS_KEY_ID=${PROD_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${PROD_AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=ap-northeast-1 bash ./ci/terraform-build.sh env/prod company01-prod-terraform