Skip to content

Commit d9c7536

Browse files
jrrudgejrrudge
authored
Merge pull request #13 from mcnamarabrian/jrrudge/pipeline
Jrrudge/pipeline
2 parents a4147b7 + 391f2c2 commit d9c7536

File tree

7 files changed

+522
-262
lines changed

7 files changed

+522
-262
lines changed

‎buildspec.yml‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
1-
version: 0.0
2-
containers:
3-
LambdaFunctions:
4-
phases:
5-
during_build:
6-
commands:
7-
- cd src/data-processor-1 && pwd && npm install async marked
8-
- cd src/data-processor-2 && npm install async marked && ls
9-
- aws cloudformation package --template $INPUT_FILE --s3-bucket $S3_BUCKET --output-template post-sam.yaml
10-
artifacts:
11-
files:
12-
- post-sam.yaml
1+
version: 0.2
2+
3+
phases:
4+
install:
5+
runtime-versions:
6+
python: 3.7
7+
commands:
8+
- pip install --upgrade aws-sam-cli
9+
build:
10+
commands:
11+
- sam build --use-container
12+
post_build:
13+
commands:
14+
- sam package --output-template-file $SAM_OUTPUT_TEMPLATE --s3-bucket $ARTIFACT_BUCKET
15+
artifacts:
16+
files:
17+
- $SAM_OUTPUT_TEMPLATE

‎img/lambda-refarch-fileprocessing-simple-pipeline.png‎

92.8 KB
Loading

‎pipeline/README.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# Serverless Reference Architecture: Real-time File Processing Deployment Pipeline
2+
3+
The Real-time File Processing reference pipeline architecture is an example of using basic CI/CD pipeline using the AWS fully managed continuous delivery service [CodePipeline](https://aws.amazon.com/codepipeline/) in order to deploy a Serverless application. Our pipeline consists of source, build and deployment stages.
4+
We use exactly the same method as in the manual deployment however we utilise [CodeBuild](https://aws.amazon.com/codebuild/) to build and package our application and the native CodePipeline CloudFormation support to deploy our package.
5+
6+
## CI/CD Pipeline Diagram
7+
8+
9+
![Reference Architecture - Real-time File Processing CI/CD Pipeline](../img/lambda-refarch-fileprocessing-simple-pipeline.png)
10+
11+
12+
## Pipeline Components
13+
14+
15+
### CloudFormation Template
16+
17+
18+
pipeline.yml is a CloudFormation template that will deploy all the required pipeline components. Once the stack has deployed the Pipeline will automatically execute and deploy the Serverless Application. See getting started for information on how to deploy the template.
19+
20+
21+
#### Deployed Resources
22+
23+
24+
* Pipeline S3 bucket, used to store pipeline artefacts that are passed between stages.
25+
* CodePipeline
26+
* CodeBuild Project
27+
* Roles for CodePipeline, CodeBuild and the CloudFormation Deployment
28+
* SNS Topic for Pipeline notifications
29+
* CloudWatch Event for Pipeline Failures
30+
31+
32+
### Source
33+
34+
35+
For this application we are hosting our source code in GitHub. Other [Source Integrations](https://docs.aws.amazon.com/codepipeline/latest/userguide/integrations-action-type.html#integrations-source) are available however this template focuses on GitHub. Whenever an update is pushed to the GitHub branch being
36+
monitored (default: master) our pipeline will begin executing. The source stage will connect to GitHub using the credentials provided and clone the branch into our pipeline artefact bucket for use in the other stages.
37+
38+
39+
### Build
40+
41+
42+
In order to run our SAM build and SAM package commands we are using [CodeBuild](https://aws.amazon.com/codebuild/), a fully managed continuous integration service. Codebuild allows us to perform a sequence of commands that we define in the [buildSpec.yml](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html)
43+
file that will execute inside the [build environment](https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html) we define using a docker container. For this project we are using the Amazon Linux 2 version 1.0 container with Python 3.7.
44+
45+
Within the buildspec.yml we are:
46+
47+
* Updating SAM to the latest version
48+
* Running SAM build as per the manual deployment
49+
* Running SAM Package again as per the manual deployment steps
50+
* Instructing CodeBuild to pass the output template back to the Pipeline for use in the deployment stage.
51+
52+
53+
54+
### Deploy
55+
56+
57+
To deploy our application stack we are not using SAM Deploy, CodePipeline doesn't support SAM natively so instead we are opting to use the CodePipeline native support for CloudFormation to deploy the template that SAM creates. The pipeline has a role it use with appropriate permissions to deploy the template created by the SAM package step which will create a stack containing the resources defined in our SAM Template. We are using [change sets](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets.html) and [approval actions](https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals-action-add.html) to demonstrate a manual approval workflow.
58+
59+
You will need to approve the deployment before the pipeline execution actually deploys any resources. Once approved, additional resources will be deployed as per the main architecture documentation.
60+
61+
62+
63+
## Getting started
64+
65+
66+
To get started using the template found in this repository under pipeline/pipeline.yaml. You will need to provide additional information to deploy the stack.
67+
68+
* GitHubToken: GitHub OAuthToken with access to be able to clone the repository. You can find more information in the [GitHub Documentation](https://github.com/settings/tokens)
69+
* AlarmRecipientEmailAddress: You will need to provide an email address that can be used for configuring notifications
70+
71+
Optionally, if you are deploying from your own repository you will need to also provide:
72+
73+
* GitHubRepoName: The name of the GitHub repository hosting your source code. By default it points to the aws-samples repo.
74+
* GitHubRepoBranch: The GitHub repo branch code pipeline should watch for changes on. This defaults to master, but any branch can be used.
75+
* GitHubRepoOwner: the GitHub repository owner. e.g. aws-samples
76+
77+
78+
79+
### Deploying the template
80+
81+
82+
You can deploy the template using either the [AWS Console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html) or the [AWS CLI](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-cli-creating-stack.html)
83+
84+
**[TODO]** Insert quick link to create CFN stack
85+
86+
87+
88+
##### Example CLI Deployment
89+
90+
91+
> aws cloudformation deploy --template-file pipeline/pipeline.yaml --stack-name "lambda-file-refarch-pipeline" --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" --parameter-overrides GitHubToken="**{replace with your GitHub Token}**" AlarmRecipientEmailAddress="**{replace with your admin email}**"
92+
93+
94+
95+
### Deploying twice for a Development and Production example.
96+
97+
98+
You can actually deploy the pipeline twice to give two separate environments. Allowing you to create a simple dev to production workflow.
99+
100+
This will allow you to build your application in your development branch and any changes will automatically be picked up and deployed by the pipeline. Once you have tested and are happy the changes can be merged to master and they will be automatically built and deployed to production.
101+
102+
Deploy the first stack using a stack name of "lambda-file-refarch-pipeline-dev" update the **AppName** parameter to be environment specific. e.g. "lambda-file-refarch-dev" and make sure to update the branch to the development one.
103+
104+
##### Example CLI Deployment for development pipeline
105+
106+
107+
> aws cloudformation deploy --template-file pipeline/pipeline.yaml --stack-name "lambda-file-refarch-pipeline-dev" --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" --parameter-overrides AppName="lambda-file-refarch-dev" GitHubToken="**{replace with your GitHub Token}**" AlarmRecipientEmailAddress="**{replace with your admin email}**" GitHubRepoBranch="develop"
108+
109+
110+
Once that has deployed and the application stack has also successfully deployed you can provision the production pipeline stack.
111+
112+
113+
> aws cloudformation deploy --template-file pipeline/pipeline.yaml --stack-name "lambda-file-refarch-pipeline-prod" --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" --parameter-overrides AppName="lambda-file-refarch-prod" GitHubToken="**{replace with your GitHub Token}**" AlarmRecipientEmailAddress="**{replace with your admin email}**" GitHubRepoBranch="master"
114+
115+
116+
##### Approval Actions
117+
118+
Any deployments will require the approval of a change set before the deployment can proceed. There will be an email sent to the admin email address, which will include a link to the approval request. You will need to ensure you have confirmed the subscription in order to receive the notification.
119+
120+
Alternatively you can do this by navigating the console or you can use the cli [Approve or Reject an Approval Action in CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals-approve-or-reject.html)
121+
122+
To use the CLI it requires the creation of a JSON document and knowing the token for the last execution. See the documentation above for details on this.
123+
124+
> aws codepipeline put-approval-result --cli-input-json file://approvalstage-approved.json
125+
126+
## Clean-up
127+
128+
In order to remove all resources created by this example you will first need to make sure the 3 S3 buckets are empty.
129+
130+
* Pipeline artefact bucket
131+
* Application input bucket
132+
* Application conversion bucket
133+
134+
Once that is complete you can remove both the Application Stack and the Pipeline Stack.
135+
Note that the pipeline stack should not be removed until the application stack has successfully deleted as it is deployed using a role present in the pipeline stack. This role is used to also delete the stack.
136+
137+
Additionally there will be some Codebuild logs and Log Groups left over in CloudWatch, these can be deleted.
138+
139+
Alternatively you can use the script /pipeline/cleanup.sh
140+
141+
Things to note:
142+
143+
* Script will remove only stacks deployed as described in the examples.
144+
145+
* Both the application and the pipeline stacks will be removed.
146+
147+
* JQ needs to be installed in order to empty the pipeline bucket as versioning is enabled. The command to delete versions and markers requires it.

‎pipeline/cleanup.sh‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/bin/bash
2+
3+
command -v jq >/dev/null 2>&1 || { echo >&2 "jq is required but it's not installed. Aborting."; exit 1; }
4+
5+
echo "Clearing out resources of lambda-file-refarch and Pipeline stacks..."
6+
echo
7+
echo "Cleaning up Application S3 buckets..." && for bucket in InputBucket ConversionTargetBucket; do
8+
echo "Clearing out ${bucket}..."
9+
BUCKET=$(aws cloudformation describe-stack-resource --stack-name lambda-file-refarch-app --logical-resource-id ${bucket} --query "StackResourceDetail.PhysicalResourceId" --output text)
10+
aws s3 rm s3://${BUCKET} --recursive
11+
echo
12+
done
13+
14+
echo "Cleaning up Pipeline S3 buckets..."
15+
BUCKET=$(aws cloudformation describe-stack-resource --stack-name lambda-file-refarch-pipeline --logical-resource-id "PipelineBucket" --query "StackResourceDetail.PhysicalResourceId" --output text)
16+
17+
echo
18+
19+
echo "Removing all versions from ${BUCKET}"
20+
21+
VERSIONS=`aws s3api list-object-versions --bucket $BUCKET | jq '.Versions'`
22+
MARKERS=`aws s3api list-object-versions --bucket $BUCKET | jq '.DeleteMarkers'`
23+
let COUNT=`echo $VERSIONS | jq 'length'`-1
24+
25+
if [ $COUNT -gt -1 ]; then
26+
echo "removing files from bucket"
27+
for i in $(seq 0 $COUNT); do
28+
KEY=`echo $VERSIONS | jq .[$i].Key | sed -e 's/\"//g'`
29+
VERSIONID=`echo $VERSIONS | jq .[$i].VersionId | sed -e 's/\"//g'`
30+
CMD="aws s3api delete-object --bucket $BUCKET --key $KEY --version-id $VERSIONID"
31+
echo ${CMD}
32+
$CMD
33+
done
34+
fi
35+
36+
let COUNT=`echo $MARKERS |jq 'length'`-1
37+
38+
if [ $COUNT -gt -1 ]; then
39+
echo "removing delete markers"
40+
41+
for i in $(seq 0 $COUNT); do
42+
KEY=`echo $MARKERS | jq .[$i].Key | sed -e 's/\"//g'`
43+
VERSIONID=`echo $MARKERS | jq .[$i].VersionId | sed -e 's/\"//g'`
44+
CMD="aws s3api delete-object --bucket $BUCKET --key $KEY --version-id $VERSIONID"
45+
echo ${CMD}
46+
$CMD
47+
done
48+
fi
49+
50+
echo "Deleting lambda-file-refarch-app CloudFormation stack..." && aws cloudformation delete-stack \
51+
--stack-name lambda-file-refarch-app
52+
53+
echo "Waiting for stack deletion..." && aws cloudformation wait stack-delete-complete \
54+
--stack-name lambda-file-refarch-app
55+
56+
echo "Deleting lambda-file-refarch-pipeline CloudFormation stack..." && aws cloudformation delete-stack \
57+
--stack-name lambda-file-refarch-pipeline
58+
59+
echo "Waiting for stack deletion..." && aws cloudformation wait stack-delete-complete \
60+
--stack-name lambda-file-refarch-pipeline
61+
62+
echo "Clearing out Application CloudWatch Log Groups..." && for log_group in $(aws logs describe-log-groups --log-group-name-prefix /aws/lambda/lambda-file-refarch-app- --query "logGroups[*].logGroupName" --output text); do
63+
echo "Removing log group ${log_group}..."
64+
aws logs delete-log-group --log-group-name ${log_group}
65+
echo
66+
done
67+
68+
echo "Clearing out CodeBuild CloudWatch Log Groups..." && for log_group in $(aws logs describe-log-groups --log-group-name-prefix /aws/codebuild/lambda-file-refarch-app-build --query "logGroups[*].logGroupName" --output text); do
69+
echo "Removing log group ${log_group}..."
70+
aws logs delete-log-group --log-group-name ${log_group}
71+
echo
72+
done

‎pipeline/main.yaml‎

Lines changed: 0 additions & 68 deletions
This file was deleted.

0 commit comments

Comments
 (0)