Lead Data Engineer
Exp: 5 - 7 yrs.
Work Location: Hyderabad
No. of positions: 1
Foundation AI is seeking a skilled Lead Data Engineer to design and develop scalable data systems and pipelines. This role involves managing large volumes of structured and unstructured data, ensuring data quality and integrity, and optimizing performance across various platforms.
As a Lead Data Engineer, you will be responsible for creating robust data pipelines, developing and optimizing data models, and ensuring the technical alignment with business objectives. You will manage configurations for tools and databases, oversee performance tuning, and collaborate closely with cross-functional teams to deliver high-quality solutions. Your role will also involve promoting best practices, conducting technical walk-throughs, and driving continuous improvement within the data team.
Architect and Design Data Systems: Design and develop scalable data pipelines and systems to manage large volumes of structured and unstructured data.
Data Pipeline Development: Develop scalable end-to-end ETL (Extract, Transform, Load) data pipelines to ingest data from various diverse sources, perform transformations, and load data into target systems to meet both functional and non-functional requirements.
Data Modeling and Optimization: Develop and optimize data models (including relational Data) for efficient storage and retrieval, ensuring data quality and integrity.
Technical Oversight: Manage technical scope, ensuring alignment with business objectives and delivering high-quality solutions. Provided required technical guidance and oversight to various cross-functional teams and team members on databases like PostgreSQL/ Vector DB and various queries, functions, and datasets.
Configuration Management: Manage the configurations for various tools and databases like postgresDB, vectorDB, etc. for both on-premises and cloud platforms for optimized performance and redundancy
Performance Tuning: Monitor and optimize data pipelines, SQL/queries, functions, procedures, and triggers for performance and scalability.
Stakeholder Collaboration: Work closely with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver solutions ensuring alignment with organizational goals.
Promoting Best Practices: Define and promote reusable, scalable, and maintainable solutions, emphasizing software engineering best practices and continuous improvement.
Communication: Communicate effectively at all levels about the importance of solution design, conducting technical walk-throughs to ensure a clear understanding of system architecture.
Continuous Improvement: Work with data and analytics experts to enhance functionality in data systems and products, driving improvements and growth within the data team.
Cloud Experience: Experience of working with AWS RDS databases is preferred
5-7 years in High-Performance Data Products or Data Systems as a Lead Data Architect/Engineer.
Technical Skills: Proficiency in orchestration tools like Airflow. Proficiency in SQL and relational databases (e.g., PostgreSQL, Aurora) and NoSQL databases (e.g., MongoDB, Cassandra).
Good knowledge of Vector database like pgvector etc.
Hands-on experience with cloud platforms (e.g., AWS, Azure, GCP) and containerization technologies (e.g., Docker, Kubernetes).
Experience with agile development methodologies (e.g., Scrum, Kanban).
Software Engineering: Proficient in software engineering best practices, unit testing, integration testing, and tools like Git, Maven, and Docker.
Security and Compliance: Familiarity with security compliances and design practices.
Communication: Exceptional interpersonal, analytical, and communication skills. Ability to explain and discuss concepts with colleagues and teams effectively.
CI/CD Pipeline: Fully adhere to and promote an entire CI/CD pipeline.
API Development: Familiarity with API development and data formats like JSON/XML.
Domain Experience: Experience in the Insurance/Legal domain is a plus.
Regulatory Knowledge: Familiarity with security and privacy regulations such as GDPR and HIPAA.
Collaboration: Proven ability to collaborate effectively with cross-functional teams in a fast-paced environment.
Problem Solving: Demonstrated ability to conduct root cause analyses and drive continuous improvement initiatives.
