Building a comment system for a static site, part 2


posted | about 12 minutes to read

tags: amazon web services comments terraform hugo tutorial database design web development dynamodb

This post is part of a series:
  1. Building a comment system for a static site, part 1
  2. Building a comment system for a static site, part 2
  3. Building a comment system for a static site, part 3

In my last post, I went over the high-level overview of what I’m looking to accomplish and discussed the technologies that I’ll be leveraging. In this post, I’m going to explore how to actually get to a functional system on a basic level. This will involve creating the AWS resources, writing the code for the Lambda functions, and creating a frontend to be able to use the system.

DynamoDB

Let’s start with storing comments - after all, if you don’t have any comments at all, what’s the point of displaying them? Identifying what we need to do this is pretty straightforward: an HTML form to post comments, a Lambda function to receive and process the posted comment, and a DynamoDB table to store it in. I’ve discussed the table design already in part 1, so here’s a Terraform example of how to set it up. I’ve already evangelized about the benefits of Terraform elsewhere, so I won’t belabor the point too much - just know that I set it up this way specifically for ease of application and maintainability.

resource "aws_dynamodb_table" "comments_table" {
  name = "comments"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "post_uid"
  range_key = "sortKey"
  
  attribute {
    name = "post_uid"
    type = "S"
  }
  
  attribute {
    name = "sortKey"
    type = "S"
  }
}

Pretty simple, right? I touched on this a little bit in part 1, but part of what’s so nice about DynamoDB tables is that we don’t have to define all our fields like we would in a relational database. Instead, we just define our partition key (hash_key) and sort key (range_key) and then we’re all set.

Lambda and IAM

To read and write from the database, as discussed, we’ll use Lambda functions. Lambda functions already load the AWS SDK for Javascript, so we don’t have to do anything fancy - just create a new function. Let’s start with writing comments to the database - what’s the point of retrieving data if there’s nothing there to retrieve, after all? Given that, we’ll create a Lambda function to accept JSON input and write it to the table. Maybe something like this:

var AWS = require("aws-sdk"); // This is included in Lambda functions, so no need for any extra installs

AWS.config.update({
	 region: "us-east-1" // or whatever your region is
});

exports.handler = function(event,context,callback) {
  const origin = event.headers.Origin || event.headers.origin; // Where is the originating request coming from? You could do some limited validation here, but mostly we use it for the header in the response

  var body = JSON.parse(event.body); //incoming JSON string - passed from the frontend, perhaps as AJAX POST
  
  // DynamoDB
  var dynamoClient = new AWS.DynamoDB.DocumentClient();
  var ts = Date.now();
  var params = {
    TableName: 'comments', // Where we're writing it
    Item: { // What we're actually writing
      'author': body.author,
      'text': body.comment_text,
      'ts': ts,
      'post_uid': body.uid,
      'sortKey': ts
    }
  };
  dynamoClient.put(params, function(err, data) { //Actually write it to the table
    if(err) {
      console.log('Error: ', err);
    } else {
      var response = {
        "isBase64Encoded": false,
        "headers": { "Content-Type": "application/json", "Access-Control-Allow-Origin": origin },
        "statusCode": 200,
        "body": "{\"result\": \"Success\"}"
      }; // This is what gets sent back
      callback(err, response); // and this actually does send it back
    }
  });
}; //If something goes wrong or if we don't return anything, the function will automatically return a 502 error code.

Again, we’re keeping things very simple here. There’s no validation, nothing but “get comment, write comment”. The weird thing that might jump out at you here, though, is that I’m not passing any database credentials or anything. Weird, right? Well, that’s the neat thing about being on AWS - we can just use roles and policies in the AWS environment to manage access to our resources. In this case, we need an IAM policy to allow the function to write data to our DynamoDB table. I’ll skip the intermediate steps here and just give some more Terraform, to be used with a zip file containing the Javascript above:

resource "aws_iam_role" "post_lambda_role" {
  name = "comments-post-lambda-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_cloudwatch_log_group" "post_lambda_logs" {
  name = "/aws/lambda/comments-post"
  retention_in_days = 14
}

resource "aws_iam_policy" "post_lambda_policy" {
  name = "post_lambda"
  path = "/"
  
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "logs:PutLogEvents",
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:REGION:ACCOUNT:table/comments",
                "arn:aws:logs:REGION:ACCOUNT:log-group:/aws/lambda/comments-post:*"
            ]
        }
    ]
}
  
  
EOF
}

resource "aws_iam_role_policy_attachment" "post_lambda_attachment" {
  role = aws_iam_role.post_lambda_role.name
  policy_arn = aws_iam_policy.post_lambda_policy.arn
}

resource "aws_lambda_function" "post_lambda" {
  filename      = "comments_post_payload.zip"
  function_name = "comments-post"
  role          = aws_iam_role.post_lambda_role.arn
  handler       = "exports.handler"
  source_code_hash = "${filebase64sha256("comments_post_payload.zip")}"

  runtime = "nodejs10.x"
}

You’ll notice I’ve been very specific about what permissions I’m granting - our Lambda function can only write to the DynamoDB table, not read, not do anything else. The least privilege principle is super important - if your function doesn’t need a permission to do what it needs to do, it shouldn’t have that permission. We can create similarly restricted policy sets for our function to retrieve comments.

Speaking of retrieving comments, let’s think about how we want to accomplish that! Same thing, really - just retrieving, instead of sending, data. I’m setting it up as a GET with a query string, and I’ll explain in the comments below where that comes in - but honestly, the code is fairly self explanatory.

// Parameters: uid

// Get all from DynamoDB
// Sort key: timestamp
// Partition key for table: uid

var AWS = require("aws-sdk");

AWS.config.update({
	 region: "us-east-1"
});

const dynamoClient = new AWS.DynamoDB.DocumentClient();

exports.handler = async (event,context) => {
  const origin = event.headers.Origin || event.headers.origin;
    var params = {
	    TableName: "comments",
	    KeyConditionExpression: "post_uid = :uid and begins_with(sortKey, :uid) ",
      ScanIndexForward: false, // This determines which directions results are returned on - either way, it'll sort on the sort key, but this determines ascending or descending order.
	    ExpressionAttributeValues: {
	      ":uid": event.queryStringParameters.uid // Here's that query string I was talking about. You can just grab it from the calling event.
	    }
    };
    var comments = await retrieveComments(params);
  }
  var response = {
          "isBase64Encoded": false,
          "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": origin},
          "statusCode": 200,
          "body": JSON.stringify(comments)
  };
  console.log(response);
  return response;
};
    
async function retrieveComments(params) {
  try {
    let data = await dynamoClient.query(params).promise();
    return data.Items;
  } catch(err) {
    console.log(err);
  }
}

You’ll notice I do some async/await stuff here. That’s because here, you need to make sure all your queries actually complete before returning data to the website - you don’t want to return an empty comments object if there are actually comments in DynamoDB. This will become more important later on in the process when we start integrating more features, but it’s a good habit to get into now.

This function can be deployed exactly the same way as the first one - just modify the IAM policy to read data instead of writing it. Once that’s done, the back-end stuff is basically in place - just one more step to go.

API Gateway

To make these functions easily web-accessible, we need to throw them behind an API Gateway - in fact, I wrote the functions with this in mind (this is how I was able to do the magic with event.queryStringParameters.uid, for example - API Gateway bundles up the incoming requests and passes Lambda a JSON object to parse). This sounds intimidating in practice, but once again, Terraform can really simplify our process by making the deployment one-click. The following code block deploys an API Gateway API with routes to both the post and get lambda functions, and adds an OPTIONS method to make sure CORS preflight requests are approved.

resource "aws_api_gateway_stage" "comments_api_prod_stage" {
  stage_name = "prod"
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  deployment_id = aws_api_gateway_deployment.comments_api_deployment.id
}

resource "aws_api_gateway_deployment" "comments_api_deployment" {
  depends_on = [aws_api_gateway_integration.post_options_integration, aws_api_gateway_integration.post_post_integration, aws_api_gateway_integration.get_get_integration, aws_api_gateway_integration.get_options_integration]
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  stage_name = "prod"
}

resource "aws_api_gateway_rest_api" "comments_api" {
  name = "comments-api"
}

resource "aws_api_gateway_resource" "post_resource" {
  path_part = "post"
  parent_id = aws_api_gateway_rest_api.comments_api.root_resource_id
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
}

resource "aws_api_gateway_method" "post_options_method" {
  rest_api_id  = aws_api_gateway_rest_api.comments_api.id
  resource_id  = aws_api_gateway_resource.post_resource.id
  http_method  = "OPTIONS"
  authorization = "NONE"
}

resource "aws_api_gateway_method_response" "post_options_response" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.post_resource.id
  http_method = aws_api_gateway_method.post_options_method.http_method
  status_code = 200

  response_models = {
    "application/json" = "Empty"
  }

  response_parameters = {
    "method.response.header.Access-Control-Allow-Headers" = true
    "method.response.header.Access-Control-Allow-Methods" = true
    "method.response.header.Access-Control-Allow-Origin"  = true
  }
}

resource "aws_api_gateway_integration" "post_options_integration" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.post_resource.id
  http_method = aws_api_gateway_method.post_options_method.http_method
  type = "MOCK"

  request_templates = {
    "application/json" = <<EOF
{
  "statusCode": 200
}
EOF
  }
}

resource "aws_api_gateway_integration_response" "post_options_integration_response" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.post_resource.id
  http_method = aws_api_gateway_method.post_options_method.http_method
  status_code = aws_api_gateway_method_response.post_options_response.status_code

  response_parameters = {
    "method.response.header.Access-Control-Allow-Headers" = "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
    "method.response.header.Access-Control-Allow-Methods" = "'OPTIONS,POST'"
    "method.response.header.Access-Control-Allow-Origin"  = "'*'" # We can restrict this one later, if we want to - for now, for testing, nah.
  }
}

resource "aws_api_gateway_method" "post_post_method" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.post_resource.id
  http_method = "POST"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "post_post_integration" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.post_resource.id
  http_method = aws_api_gateway_method.post_post_method.http_method
  content_handling = "CONVERT_TO_TEXT"
  integration_http_method = "POST"
  type = "AWS_PROXY"
  uri = "arn:aws:apigateway:${var.region}:lambda:path/2015-03-31/functions/${data.aws_lambda_function.post_lambda.arn}/invocations"
}

resource "aws_lambda_permission" "execute_apigw_lambda_post" {
  statement_id = "AllowExecutionFromAPIGateway"
  action = "lambda:InvokeFunction"
  function_name = data.aws_lambda_function.post_lambda.function_name
  principal = "apigateway.amazonaws.com"
}

resource "aws_api_gateway_resource" "get_resource" {
  path_part = "get"
  parent_id = aws_api_gateway_rest_api.comments_api.root_resource_id
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
}

resource "aws_api_gateway_method" "get_options_method" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.get_resource.id
  http_method = "OPTIONS"
  authorization = "NONE"
}

resource "aws_api_gateway_method_response" "get_options_response" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.get_resource.id
  http_method = aws_api_gateway_method.get_options_method.http_method
  status_code = 200

  response_models = {
    "application/json" = "Empty"
  }

  response_parameters = {
    "method.response.header.Access-Control-Allow-Headers" = true
    "method.response.header.Access-Control-Allow-Methods" = true
    "method.response.header.Access-Control-Allow-Origin"  = true
  }
}

resource "aws_api_gateway_integration" "get_options_integration" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.get_resource.id
  http_method = aws_api_gateway_method.get_options_method.http_method
  type = "MOCK"

  request_templates = {
    "application/json" = <<EOF
{
  "statusCode": 200
}
EOF
  }
}

resource "aws_api_gateway_integration_response" "get_options_integration_response" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.get_resource.id
  http_method = aws_api_gateway_method.get_options_method.http_method
  status_code = aws_api_gateway_method_response.get_options_response.status_code

  response_parameters = {
    "method.response.header.Access-Control-Allow-Headers" = "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
    "method.response.header.Access-Control-Allow-Methods" = "'OPTIONS,GET'"
    "method.response.header.Access-Control-Allow-Origin"  = "'*'" # We can restrict this one later, if we want to - for now, for testing, nah.
  }
}

resource "aws_api_gateway_method" "get_get_method" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.get_resource.id
  http_method = "GET"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "get_get_integration" {
  rest_api_id = aws_api_gateway_rest_api.comments_api.id
  resource_id = aws_api_gateway_resource.get_resource.id
  http_method = aws_api_gateway_method.get_get_method.http_method
  integration_http_method = "POST"
  content_handling = "CONVERT_TO_TEXT"
  type = "AWS_PROXY"
  uri = "arn:aws:apigateway:${var.region}:lambda:path/2015-03-31/functions/${data.aws_lambda_function.get_lambda.arn}/invocations"
}

resource "aws_lambda_permission" "execute_apigw_lambda_get" {
  statement_id = "AllowExecutionFromAPIGateway"
  action = "lambda:InvokeFunction"
  function_name = data.aws_lambda_function.get_lambda.function_name
  principal = "apigateway.amazonaws.com"
}

Once you’ve applied this, in addition, you’ll have to log into the AWS console, go to the API Gateway resources that was just created, and toggle off, and then back on, the “Use Lambda Proxy integration” checkbox under the “Integration Request” section of both the POST and GET methods of the API Gateway resource. This is to get the method response to show up as “Proxy”. I tried for quite some time to just get this working through Terraform with no success - so unfortunately, a slight manual step.

What the heck this is complicated

Now, at this point you may be thinking “what happened to ‘just the basics’, Alli? That last snippet was like 200 lines of Terraform!”, and that is an extremely fair question! The answer is that even a basic setup for something like this really does require a certain amount of things be built out in AWS to support it from an infrastructure perspective. Sure, you could just run Node on an EC2 server or whatever, but that kind of defeats the purpose of going serverless - so instead, we can build out all this infrastructure now, and then it’s sustainable and defined in an easy way for us to keep up to date later. (The other half of the argument is that a lot of this would be necessary in some form anyway - half of the stuff above is related to the OPTIONS method needing to work properly, and that would have had to be done no matter where it was hosted.)

Frontend

The good news, though, is that…that’s it. You’re done. If you were to fire up Postman and start firing requests against your API endpoint you absolutely could, and it should work. All that remains is to write the frontend - of course, the specific implementation will be different for everyone depending on your use case, but here’s a sample basic implementation using jQuery and jquery-dateFormat, and including basic Bootstrap 4 styling. I assume a <div> element with id comments. (Alternatively, you can see a sample implementation written in vanilla JavaScript in this page’s source).

$(document).ready(function() {
  var comments_text = '';
  $.get('http://endpoint.apigateway.amazonaws.com/get?uid={{ .File.UniqueID }}', function(comments) { //In Hugo, .File.UniqueID is an md5 hash, and I am using it in a partial page template. Implementation for this will vary based on platform.
    if(comments.length <= 0) {
      comments_text = '<div class="text-muted">No comments found!</div>';
    } else {
      $.each(comments, function(i, commentObject) {
        var formattedDate = "posted " + $.format.date(commentObject.ts, "MMM d, yyyy") + " at " + $.format.date(commentObject.ts, "h:mm a");
        comments_text += '<div class="card card-body mb-1"><div class="comment-author">' + commentObject.author + '</div><div class="comment-body">' + commentObject.text + '</div><div class="comment-timestamp text-muted">' + formattedDate + '</div></div>'; //css classes
      });
    }
    comments_text += '<div class="comment-form"><form class="form-horizontal">' +
      '<div class="form-group"><label for="author-{{ .File.UniqueID }}" class="control-label">Your name:</label><input class="name form-control" id="author-{{ .File.UniqueID}}" maxlength="20" placeholder="Limit 20 characters" /></div>' +
      '<div class="form-group"><label for="text-{{ .File.UniqueID }}" class="control-label">Comment text:</label><textarea class="text form-control" id="text-{{ .File.UniqueID}}" maxlength="1000" placeholder="Limit 1000 characters."></textarea></div>' +
      '<button class="form-submit btn btn-success">Post</button></form></div>';
      
    $('#comments').html(comments_text);
  })
  .fail(function() {
    $('#comments').html('<div class="text-muted">Comments could not be loaded.</div>');
  });
})

.on('click', '.form-submit', function(event) {
  event.preventDefault();
  $(this).prop('disabled', true);
  var wholeForm = $(this).parent();
  wholeForm.find('.form-control').prop('disabled', true);
  var formdata = {"author": $('#author-{{ .File.UniqueID }}').val(), "comment_text": $('#text-{{ .File.UniqueID }}').val(), "uid": "{{ .File.UniqueID}}" }; // Just like earlier, make sure to use something here that's applicable to your environment.
  
  $.ajax({
    type: "POST",
    url: 'http://endpoint.apigateway.amazonaws.com/post/',
    dataType: "json",
    crossDomain: "true",
    contentType: "application/json; charset=utf-8",
    data: JSON.stringify(formdata),
	
    success: function(data) {
      if(data.result == "Success") {
        wholeForm.html('<div class="card card-body mb-1"><div class="comment-author">' + formdata.author + '</div><div class="comment-body">' + formdata.comment_text + '</div><div class="comment-timestamp text-muted">posted ' + $.format.date(Date.now(), "MMM d, yyyy") + " at " + $.format.date(Date.now(), "h:mm a") + '</div></div>');
      }
	  },
	  error: function(jqXHR) {
	  
    }
  });
});

Again, this is super basic. I set up some classes that you could use to do CSS styling, but you’ll have to do that yourself. That said, this…pretty much handles our MVP. You can see the comments on a page (based on the page ID) and post your own comments - and you can probably see where some of this stuff is happening if you inspect the code on this page. You’ll notice a lot of similarities.

This is great, but there’s still work that can be done here. Our “nice to haves” are still out there, for one thing. Next post, I’ll dive into how to build on this and bolt on some more features - or, if you’d like to skip ahead in the reading, please feel free to check out the GitHub repository with the server-side JavaScript.