Microservices - The need to move towards serverless architecture

The past decade has been extremely eventful in the technology space. It seems to have reached a state of exponential growth. Lots of innovation and lots of consolidation. The architecture that was once considered the holy grail (the layered architecture) doesn't necessarily scale up in the world of Polyglot programming. The Microservices have now almost become the default design pattern. Your service needs to do one thing good and should be accessible through well defined APIs (API economy anyone!!). As such it's perfectly OK to have services in different process space, different technology, different storage and even different Operating system. After all a recommendation engine driven by a GraphDB will most certainly outperform a normal RDBMS driven engine.

At our organization, the move towards microservice based architecture started 6 months back. Things are good until you run into issues that are more operational. In a monolithic application, we used to have one server application with a Central DB repository. As long they were monitored , we would have a decent sleep at night. In the world of microservices that complexity of managing and monitoring increases manifold. First you need to monitor individual services. Then you need to monitor the inter-service communication. And how do you scale in or out the more busy and labor intensive services? Service management and monitoring is probably the single most important thing that needs to be considered while moving to a Microservice Based architecture.

A serverless architecture helps us addressing some of these issues. Serverless necessarily doesn't mean you don't need a server to run your application. It simply means you are abstracted from managing, monitoring and scaling the services. In a Cloud driven world, this is somewhat similar to a PaaS offering. However PaaS doesn't lend itself very well when you have 100's of services. You would probably have to club a bunch of those in a container and deploy as a Unit. The core problem of managing and monitoring services remain the same.

Last year at AWS summit, Amazon unveiled AWS Lambda where a component can be invoked on the fly on an event trigger. Recently, Microsoft announced about Azure functions which is very similar. In my opinion this somewhat addresses the issues associated with Microservice management. The Lambda functions can each be a Microservice. We could have the AWS API gateway accepting requests from the client. Lambdas can be invoked directly when the API receives a request and shut down immediately after doing the job. Lambdas can also react to Kinesis Stream, S3 object puts or DynamoDB stream. The good thing is each will be scaled up or down as the need be. The cloudwatch metrics monitors each of the services without you having to write any infrastructure level code.

Another approach is to leverage the modern Databases. RethinkDB, Firebase, DynamoDB and PostGres are now more than happy to expose their transaction logs in realtime to anyone who wants to consume (Something similar to CDC which I think was an afterthought). They also expose APIs directly where a client can invoke and consume. RethinkDB also pushes the details back to the client on any commits (rather than client polling for it). Though I am not in favor of such an approach, it works perfectly well for prototypes and small applications.

As with everything, there are quite few things to consider before you move to this approach. There are many quirks that you would face along the way. We hit quite a few road-blocks (especially on security and AWS Lambda) during our Pilot. Shall share the details in a later post. Have a great week!!