Neelima Reddy

Microsoft Certified Data Engineer, specializing in Big Data, Hadoop ecosystems, databases, data warehousing, and Azure-related technologies. Demonstrated expertise in big data analytics, proficiently utilizing Python and Scala programming languages, the Spark Framework, Azure cloud services, and a range of Hadoop eco-system tools. A skilled data professional, proficient in Data Engineering, excelling in managing end-to-end ETL data pipelines and addressing intricate architectural and scalability challenges. Skilled in SQL queries, encompassing DDL, DML, and diverse database objects, with hands-on experience in various file formats including ORC, Parquet, and CSV. Extensive proficiency in utilizing various Azure cloud services both IAAS and PAAS, including Blob storage account, Key Vault, Logic Apps, Function Apps, and more for building pipelines, analytics, storage, networking, security, grouping, resource management and other purposes. Used Azure stream analytics and Event hubs for processing continuous data from IoT devices, social media platforms, and applications to get valuable insights. Highly capable of establishing CI/CD frameworks for data pipelines with tools such as Jenkins, ensuring streamlined automation and deployment. Extensive experience with T-SQL in constructing Triggers and tables, implementing stored Procedures, Functions, Views, User Profiles, Data Dictionaries and Data Integrity. Good Knowledge of libraries like NumPy, Pandas, SciPy, TensorFlow, Scikit-learn, and Matplotlib in Python. Firm experience in Hadoop architecture and various tools including HDFS, Yarn, MapReduce, Apache spark, Sqoop, Flume, Zookeeper, Hive, Pig, HBase, Kafka, Oozie, etc.,  Leveraged Azure Data Factory to seamlessly integrate on-premises databases (MySQL, MS SQL, Cassandra) which are SQL and NO SQL and cloud-based storage (Blob storage, Azure SQL DB), applying advanced transformations, and loading data into Snowflake. In-depth, understanding of Apache spark job execution components like DAG, lineage graph, DAG Scheduler, Task scheduler, and Stages and worked on relational, NoSQL databases including HBase, Postgres SQL, Cassandra, Cosmos DB and Mongo DB. Proficient in leveraging Spark's RDDs (Resilient Distributed Datasets) and Data Frame APIs in Scala to process and analyze large datasets efficiently. • Utilized Git as a version control tool for maintaining the code repository. Actively engaged in continuous improvement initiatives, proactively recommending modifications to existing SDLC processes, resulting in enhanced efficiency and successful project execution.

Leveraged Azure Data Factory to seamlessly integrate on-premises databases (MS SQL, Cassandra) which are SQL, No SQL and cloud-based storage (Blob storage), applying advanced transformations by using Databricks, and loading data into Snowflake.
• Developed custom Python, Spark, and Bash scripts to facilitate data transformation and loading across both on-premises and cloud platforms.
• Developed and optimized ETL processes using PySpark for processing large datasets, enabling efficient data ingestion, transformation, and storage.
Integrated Spark workflows with Azure Data Lake, Blob Storage, and Azure Synapse for scalable data processing, storage, and analysis solutions.
• Created data pipelines in Azure Data Factory (ADF) using Linked Services, and Datasets to efficiently extract, transform, and load data from various sources including Azure SQL, Blob storage, Azure SQL Data Warehouse, and write-back tool.
• Managed the seamless import and export of databases using SQL Server Integration Services (SSIS) and Data Transformation Services (DTS Packages).
• Utilized Azure DevOps for managing source code, builds, releases, and infrastructure as code (IaC), leveraging tools such as ARM templates or Terraform.
• Successfully migrated applications from Cassandra DB to Azure Data Lake Storage Gen A1 using Azure Data Factory.
• Designed and implemented end-to-end IoT solutions on the Azure platform including Azure IoT Hub, Azure IoT Edge, Azure Time Series Insights, and Azure Stream Analytics, to enable seamless device-to-cloud and cloud-to-edge communication, encompassing Device Connectivity, Data Processing, storage, and insights generation.
• Developed and implemented best practices, guidelines, and standards for Data Governance, Data Quality, and Data Integration.
• Utilized networking concepts such as VPN, and firewalls to ensure secure and reliable communication in IoT environments.

MS in IT from Clark University, Massachusetts.