Our Analytics Solution frame work provides a platform to quickly build any analytical solutions. This frame work provides the basic building blocks enabling the deployment of data transformation and data science models quickly without having to worry about the needed plumbing.
Data is acquired from sources such as HMS, EMR, Public Heath Data, Lab records etc. and is fed into the pipeline. Our technology choice for the data pipeline is Luigi, while other pipelines such as AWS, Azkaban and Pinball can also be considered based on customer preferences. Luigi is preferred because of its simple code level dependency mapping eliminating the need for complex configuration files. There may be a need for an intermediate data store for the pipeline to use while it processes the data. This data store can vary based on the type and volume of data. The data pipeline orchestrates the tasks involved in cleansing and transforming the data.
Transformed data is streamed to a master store, which is immutable. Only further data addition is allowed. While multiple options are available based on the type, structure and volume of data, in this instance, Amazon S3 is our data store of choice.
The processing portion of the analytics framework is built on Lambda Architecture with two paths for the data.