Software Engineer - Cloud Data Pipelines

Job Summary (List Format):

• - Design, build, and maintain scalable data pipelines for executing large-scale R and Python scripts.
• - Develop, containerize, and deploy custom scripts to run reliably in cloud-native (serverless or containerized) environments.
• - Integrate and automate data workflows using managed cloud services on Google Cloud Platform (GCP) or Microsoft Azure.
• - Implement and manage workflow orchestration tools (e.g., Apache Airflow, Prefect) for scheduling and monitoring.
• - Ensure data quality through automated logging, error handling, and testing frameworks.
• - Optimize data processing jobs for performance, scalability, and cost-efficiency.
• - Collaborate on integrating advanced analytical models into production environments.
• - Leverage containerization (Docker, Kubernetes) and CI/CD pipelines for robust deployment.
• - Work with both relational and non-relational databases (e.g., PostgreSQL, BigQuery, Snowflake, Azure SQL).
• - Apply best practices in software engineering, including version control, automated testing, and API design.
• - (Preferred) Assist in migrating legacy analytical scripts to distributed cloud architectures.
• - (Preferred) Utilize Infrastructure as Code tools (Terraform, Bicep) for cloud resource management.