Big Data Infrastructure Engineer

Job Overview We are seeking a Big Data Infrastructure Engineer with a strong AI-first and automation-driven mindset to join our infrastructure team and support the administration and operation of enterprise Big Data platforms and their underlying infrastructure. The ideal candidate should have strong hands-on experience in Linux operating system administration and database administration, with particular focus on configuration, troubleshooting, performance tuning, optimization, monitoring, and production support. The candidate is expected to actively leverage AI-assisted tools and automation to improve troubleshooting, operational efficiency, scripting, documentation, research, and day-to-day engineering activities, while maintaining the technical judgment required to validate solutions before implementation. The role also requires practical knowledge of Docker and Kubernetes, along with a good understanding of clustering, high availability, distributed systems, and Big Data concepts. A strong willingness to continuously learn and stay current with emerging AI, automation, infrastructure, and Big Data technologies is essential. Duties and Responsibilities • Leverage AI-assisted tools to improve troubleshooting, log analysis, scripting, documentation, research, and operational efficiency while validating outputs before implementation. • Identify repetitive operational activities and develop automation solutions using Shell/Bash, Python, Ansible, APIs, or other appropriate technologies. • Continuously evaluate emerging AI and automation capabilities and identify practical opportunities to improve infrastructure operations and engineering workflows. • Administer, configure, manage, troubleshoot, and optimize Linux operating systems supporting production and non-production environments. • Monitor and analyze CPU, memory, disk, filesystem, network, processes, and system services, and perform configuration and performance tuning when required. • Administer and manage PostgreSQL, MySQL/MariaDB, and Redis, including configuration, access management, backup and recovery, monitoring, troubleshooting, maintenance, and performance optimization. • Support database replication, high availability, backup/recovery, and capacity management requirements. • Support and maintain Docker and Kubernetes environments, including deployment, configuration, monitoring, troubleshooting, scaling, and cluster administration. • Support clustered and distributed platforms with focus on high availability, replication, failover, load balancing, quorum, capacity management, and disaster recovery. • Support the installation, configuration, monitoring, administration, and upgrade of Cloudera/Hortonworks and Hadoop-based environments. • Support and troubleshoot Hadoop ecosystem components such as HDFS, YARN, Hive, Spark, HBase, and Kafka, as well as related platforms such as Airflow, Superset, and Trino/Presto where applicable. • Perform production monitoring and support using tools such as Zabbix and Grafana, and participate in incident management, root cause analysis, and corrective/preventive actions. • Support security integrations and technologies such as Ranger, LDAP, and Kerberos. • Collaborate with development, infrastructure, and other technical teams on deployments, upgrades, infrastructure changes, troubleshooting, and production support. • Maintain technical documentation, operational procedures, automation, and infrastructure configuration records. Skills and Qualifications • 3-5 years of relevant hands-on experience in Linux/System Administration, Database Administration, Big Data Infrastructure, DevOps, or a related infrastructure role. • Bachelor's Degree in Computer Science, Computer Engineering, Information Technology, or a related field. • Strong AI-first and automation-driven mindset, with demonstrated ability to use AI-assisted tools effectively in technical workflows and critically validate generated recommendations before applying them. Big Data Infrastructure Engineer - Job Description Good scripting and automation skills using Shell/Bash; knowledge of Python, Ansible, APIs, or similar technologies is highly desirable. • Strong hands-on knowledge of Linux administration, including system configuration, service management, resource management, storage/filesystems, permissions, networking, troubleshooting, and performance optimization. • Good hands-on knowledge of PostgreSQL, MySQL/MariaDB, and Redis administration, including configuration, backup and recovery, users and privileges, monitoring, maintenance, and performance tuning. • Good understanding of database concepts including connections, transactions, locks, indexing, query performance, replication, and high availability. • Good hands-on understanding of Docker and Kubernetes, including containers, images, pods, deployments, services, storage, networking, monitoring, resource management, and troubleshooting. • Good understanding of clustering and distributed system concepts, including high availability, replication, failover, load balancing, and quorum. • Good understanding of networking fundamentals, including TCP/IP, DNS, ports, routing, connectivity, and network troubleshooting. • Good understanding of Big Data concepts and the Hadoop ecosystem, with familiarity or hands-on experience in HDFS, YARN, Hive, Spark, Kafka, and HBase. • Familiarity with Cloudera or Hortonworks platforms is highly desirable. • Familiarity with Zabbix/Grafana, Ranger/LDAP/Kerberos, CI/CD tools, and Trino/Presto is an advantage. • Strong troubleshooting, analytical, and problem-solving skills, with the ability to investigate issues systematically and identify root causes. • Ability to work effectively in production environments, collaborate across technical teams, take ownership of assigned activities, and continuously develop technical knowledge. Preferred Certifications • Relevant Linux certifications such as RHCSA or RHCE. • Kubernetes certification such as CKA. • PostgreSQL or MySQL-related certifications/training. • Red Hat Ansible or other relevant automation certifications. Note: Certifications are considered an advantage and are not a substitute for practical hands-on experience.