Data Engineering has evolved constantly over the past decade. Although my exposure to this currently is limited but I never shied away from learning anything new when I get a chance. Below are the areas I have worked on:
 

• Worked on creating databases pipelines from the scratch and helped the engineering teams identify data gaps and optimize the structure. 

• Helped the team in building analytics layer on top of warehouse layer to ensure data availability for advanced analytics and reporting purposes.

• Experienced working with distributed partitioned storage systems and writing efficient codes to process big data of size up to terabytes. 

• API Integrations from 3rd party data and writing the batch process for data to be readily available in automated manner.

• Experienced in building bots to scrape & collect data from online websites (Such as e-commerce, real-estate, stock market data)

• Data validations at scale to ensure error-free data

​

​

Tools/Platform/Frameworks I have worked on:  MySQL, SQL Server, Spark, Hive,  Cosmos DB, Databricks, GBQ

Programming Languages: SQL, Python, R