The purpose of this article is to explain the architecture and the technology stack used in the creation of the Eco-system that was delivered to a Fortune 500 public retail company. Due to NDA agreements, I’m obligated to obscure the actual purposes and specific use cases of those applications.
Background
Our company contracted with this Fortune 500 public retail company company, and let’s refer to it as X from now, to build a data collection application of some type (beyond the scope of this article). That application was built in a Monolithic architecture. Meaning, One application, one project that connects all the application resources, serves the web pages, handles all the HTTP and AJAX requests. The application would also interface with the databases, caching services, ActiveDirectory, and anything else and the application may utilize. For that time and given that it was only application and we weren’t envisioning a development for a whole eco-system. And Honestly, it was my very first project in Node.js or anything for that matter and I don’t think I even know what a microservices would even mean. 🤔 😗
The first database design was consisting of about 16 tables. (I also can’t share that) but it’s worth mentioning that I designed that database as I was taking Database 1 and that was my first ever database design. I even remember I would go to class and come back and make more changes (Just like that). The current database was redesigned from the ground-up to to accommodate the new applications. It was also better normalized and optimized by the addition of multiple indexes to improve the performance of the queries. Now it’s 50 tables and 4 views.
Fast Forward ...
Apparently the company X liked what we did and they came back to us with 4 new projects and V2.0 of the the collection application with more functionality, business rule and administrative capabilities. Then we had to rethink, the architectural choices that we made before. Including the database design I was actually done with Database I and Database II then
System Architecture
The current architecture consists of 11 Microservices, connected together through a unifying RESTful API. All the applications and mobile apps communicate with the system resources and with each other through that API. The microservices, depending on their purposes and functionality, would interact with one or more of the following pieces (The database server (Microsoft SQL Server), the Caching Service/ Session Manager (Redis), ActiveDirectory (User and Domain Management), Queueing service (RabbitMQ), Machine Learning and data processing service (R servers), Reporting Service (CrystalReports) .
Server architecture
In this project, we used 6 servers; 3 development servers located and managed by our office and 3 servers for Staging and Production Servers that are managed by the client but we have access to do some debugging and deployment to their staging servers.
I was responsible for managing and development servers and configuring the Continuous Integration CI builds. And the Lead Developer was responsible for deploying our code to their staging servers.
Technology Stack:
- Node.js / Restify
- Microsoft SQL Server
- Bookshelf.js / Knex
- The micro service uses an Object Relational Mapping Module for data access to abstract away the vendor-specific SQL syntax which makes the application more portable and more easily-switchable to any SQL flavor (MSSQL, MySQL, PostgreSQL, … etc.)
- we used to use Sequelize but we switched over to bookshelf/Knex as it gives us more control over the queries and at the time it seemed to be much easier.
- The micro service uses an Object Relational Mapping Module for data access to abstract away the vendor-specific SQL syntax which makes the application more portable and more easily-switchable to any SQL flavor (MSSQL, MySQL, PostgreSQL, … etc.)
- Redis
- RabbitMQ
- DeployR / R Server
- ActiveDirectory
- Drone.io
Development Style
The development was intended to, and did start, in a Test-Driven-Development TDD. But I confess, we didn't follow through leading to a tremendous technical debt. A problem that we started to slowly reduce, by writing more unit tests and by utilizing applications such as PAW. Which allows us to create collections of HTTP requests with its parameters and run them all it once making it easy to spot any failures to quickly go fix it.
We also started a process to review and accept any features and bug fixes; the project manager is the only person who's allowed to close any issue or task item. Though this might not be the best solution, but it's working for the time being until we pay our debt.
