Express.js style flow for AWS Lambda

Note: this code was created as a proof-of-concept in a short period 😁 If you have any advices on improving it, feel free to leave comments.

Using AWS Lambda with API Gateway is amazing in many points. You get a handy environment with deployment and routing configuration if you use Serverless, automatic log collection with CloudWatch and web-based testing environment.

However, API Gateway uses its own format for input and output data which makes it impossible to reuse existing libraries.

One of the most popular HTTP frameworks for Node is Express which has thousands of plugins and libraries. So it this tutorial we are going to make them reusable inside Lambda functions.

Pre-flight

Let’s assume that you already have got your AWS environment set up. If not, you may refer to Serverless Getting Started Guide.

In order to understand what we need to do, let’s take a look at example event generated by API Gateway

API Gateway Proxy Request Event

{
  "body": "{\"test\":\"body\"}",
  "resource": "/{proxy+}",
  "requestContext": {
    // ...
    },
    "stage": "prod"
  },
  "queryStringParameters": {
    "foo": "bar"
  },
  "headers": {
    "Accept-Language": "en-US,en;q=0.8",
    "Accept-Encoding": "gzip, deflate, sdch"
  },
  "pathParameters": {
    "proxy": "path/to/resource"
  },
  "httpMethod": "POST",
  "stageVariables": {
    "baz": "qux"
  },
  "path": "/path/to/resource"
}

In general it is very similar to most Request implementations, but it is a raw object and working with it directly forces us to do many things, e. g. Case-insensitive headers getter

// Raw
event.headers['Content-Type']; // case-sensitive

// Express
req.get('content-type'); // case-insensitive

API Gateway Proxy Response Event

{
  "statusCode": 200,
  "headers": {
    "Content-Type": "application/json"
  },
  "body": "{\"response\":\"body\"}",
}

And the same about handy response methods, which Express provides

// Raw
response = {
  statusCode: 200,
  headers: {},
};
response.body = JSON.stringify({response: 'body'});
response.headers['Content-Type'] = 'application/json';

// Express
res.status(200).json({response: 'body'})

Setup

Note that AWS currently supports NodeJS 6.x LTS or lower. This tutorial is aimed to this version. You can install multiple NodeJS versions using nvm utility.

First, we need to have Express itself installed. We are not going to use the application, but only request and response classes included in the framework.

npm i --save express

And create a module file. Let’s name it http-helper.js

Our module will export a function which will accept middleware as argument and return a valid Lambda function.

module.exports = (middleware) => {
  return (event, context, callback) => {
    // Implementation
  }
}

Creating the pipeline

As we need to support multiple middleware, it is required to implement an express-compatible pipeline. For example:

const use = require('http-helper.js')

module.exports = use([
  (req, res, next) => {},
  (req, res) => {},
])

In order to do that we will use pipeworks package.

npm install —save pipeworks

We are going to use following features of it:

• const pipe = pipeworks() — creates a new pipeline instance
• pipe.fit((context, next) => { next(context) }) — appends an executor to pipeline
• pipe.fault((context, error) => {}) — defines error handler
• pipe.flow(context) — executes the pipeline

As you can see, the usage is different from Express, which passes request and response objects into middleware. Also Express doesn’t require to pass arguments into next() function. Instead a error can be thrown using it.

In order to achieve that, we will transform our middleware.

Pipeworks-to-Express

// pipeline.js
const { reduce } = require('lodash');
const pipeworks = require('pipeworks');

const Pipeline = (middleware) => {

  const pipe = pipeworks();

  reduce(middleware, (pipe, executor) => {
    return pipe.fit(
      // Pipeworks compatible definition
      (context, next) => {
        // We extract request and response from context
        executor(context.req, context.res, (err) => {
          // If error passed to next, throw it to trigger fault
          if (err) throw err
          // otherwise run next compatible to pipeworks
          else return next(context);
        })
      }
    )
  }, pipe)

  return pipe
}

module.exports = {
  Pipeline,
}

Now we inject this code into our handler.

// http-helper.js
const { Pipeline } = require('./pipeline')

module.exports = (middleware) => {
  return (event, context, callback) => {
    const req = event;
    const res = {};
    const pipe = Pipeline(middleware);

    pipe.fault((context, err) => {
        callback(err);
    })

    pipe.flow({req, res});
  }
}

Now we have a very basic implementation, but it is missing the critical part: request and response handling.

Dealing with Requests and Responses

Behavior of node http package is different in different version. This will not work with version different from 6.x

// http-internals.js
const ExpressRequest = require('express/lib/request');
const ExpressResponse = require('express/lib/response');

These are basic object for request and response, however you cannot use them directly without setup.

First issue is that those are not constructors, but prototypes, so we need to use Object.create to make a new instance (new will not work with prototypes).

Next is that they both require app property containing express application and circular references on each other (request.res and response.req).

Usage of app is limited to one method get with parameter 'etag fn' which should return tag compiler. So we can simply mock the object. While circular references can be dealt with enumerable: false option.

The next problem is with ExpressResponse which extends http.ServerResponse, but it does not call its constructor, so the response is not initialized by default and we cannot call constructor on the prototype. However, we can merge these objects.

And the last is response send method. Express is meant to be used with http connections or sockets. So by default it writes output to buffer and awaits for buffer to be flushed. This does not happen with Lambda functions, so we need to overwrite send method.

With request we only need to transform properties of event object to suitable request properties.

// http-internals.js
const { reduce } = require('lodash')
const { ServerResponse } = require('http')
const ExpressRequest = require('express/lib/request');
const ExpressResponse = require('express/lib/response');
const { compileETag } = require('express/lib/utils')

// Emulate express application settings behavior
const app = {
  get: (key) => {
    switch (key) {
      case 'etag fn': return compileETag('weak')
      default: return undefined
    }
  }
}

module.exports.Request = (event) => {
  const request = Object.create(ExpressRequest, {
    // Enumerable must be false to avoid circular reference issues
    app: { configurable: true, enumerable: false, writable: true, value: app },
    res: { configurable: true, enumerable: false, writable: true, value: {} },
  });

  Object.assign(request, {
    // HTTP Method
    method: event.httpMethod,
    // Headers converted to lowercase
    headers: reduce(event.headers, (h, v, k) => set(h, k.toLowerCase(), v), {}),
    // Path
    url: event.path || '/',
    // Route parameters
    params: event.pathParameters || {},
    // Request context
    requestContext: event.requestContext || {},
    // API Gateway resource definition
    resource: event.resource || '/',
    // Transformed query parameters
    query: event.queryStringParameters || {},
    // Stage variables
    stage: event.stageVariables || {},
    // Body
    body: event.body,
  })

  return request
}

module.exports.Response = (request) => {
  const nodeResponse = new ServerResponse(request);
  const response = Object.create(ExpressResponse, {
    // Enumerable must be false to avoid circular reference issues
    app: { configurable: true, enumerable: false, writable: true, value: app },
    req: { configurable: true, enumerable: false, writable: true, value: request },
  });

  Object.assign(response, nodeResponse);

  response.send = (body) => {
    const ret = ExpressResponse.send.call(response, body);
    for (let callback of response.outputCallbacks) {
      if (typeof callback === 'function') {
        callback();
      }
    }
    return ret;
  }

  // Convert to API Gateway object
  response.toJSON = () => {
    // If headers sent, buffer contains headers line in first index
    if (response.headersSent) delete response.output[0]
    return {
      // Response Status Code
      statusCode: response.statusCode,
      // Response Headers
      headers: reduce(
        // Take response header names
        response._headerNames,
        // Assign header values to new object using header names
        (headers, name, key) => set(headers, name, response._headers[key]),
        {}
      ),
      // Body String
      body: reduce(response.output, (body, buffer) => {
        // Buffer may be undefined
        if (buffer) {
          body += buffer.toString()
        }
        return body
      }, '')
    }
  }

  return response;
}

Executing Lambda callback

Now we put all the parts together and set up finish event of response to execute the callback.

Also we can assume that there might be single or multiple middleware passed as an argument

// http-helper.js
const { Request, Response } = require('./http-internals')
const { Pipeline } = require('./pipeline')

module.exports = (...middleware) => {
  return (event, context, callback) => {
    const req = Request(event)
    const res = Response(req)
    req.context = context
    req.res = res

    // This is required to avoid multiple callback executions.
    let finished = false

    res.on('finish', () => {
      if (finished) return
      finished = true
      callback(null, res.toJSON())
    })

    Pipeline(middleware).fault((context, error) => {
      callback(error)
    }).flow({ req, res })
  }
}

Example usage

// test-http.js
const use = require('./http-helper')

module.exports.testHttp = use(
  (req, res, next) => {
    req.middlewareOneExecuted = true
    next()
  },
  (req, res) => {
    res.send({
      m1: req.middlewareOneExecuted,
      m2: true,
    })
  }
)

Test request: