Job Description

About the company

Geekhunter is hiring on behalf of our client, a leading publicly listed property enterprise in Indonesia, which is distinguished by its robust asset base, consistent financial performance, and enduring revenue streams.

Role Summary

We are seeking a meticulous and technically proficient AI QA Engineer to uphold the quality, reliability, and ethical integrity of our Generative AI (GenAI) systems. This role is pivotal in validating AI outputs, monitoring performance indicators, and contributing to continuous system enhancement through Human-in-the-Loop (HITL) evaluations, reinforcement feedback mechanisms, and sentiment analysis. The successful candidate will bring a structured QA approach, deep technical insight, and a commitment to delivering business-aligned, responsible AI outcomes.

Job Responsibilities

GenAI Quality Assurance & Performance Monitoring

• Oversee and interpret GenAI performance dashboards to assess system health and output quality
• Identify inconsistencies, accuracy deviations, or potential bias in AI-generated responses
• Provide ongoing assessments of model performance and user experience metrics

Human-in-the-Loop (HITL) Evaluation

• Propose targeted test cases and scenarios to support HITL validation processes
• Partner with cross-functional teams to ensure human feedback is effectively integrated into the QA workflow

Reinforcement Feedback & Model Optimisation

• Analyse GenAI outputs to inform reinforcement learning strategies and prompt refinement
• Recommend enhancements to model behaviour, including adjustments to Retrieval-Augmented Generation (RAG) pipelines

Sentiment Analysis & User Feedback Insights

• Review sentiment trends across user feedback (positive, neutral, negative)
• Collaborate across departments to translate feedback into actionable system improvements

Quality Assurance & Compliance Alignment

• Ensure QA protocols adhere to internal standards, ethical AI principles, and relevant regulatory frameworks
• Maintain comprehensive documentation of QA methodologies, findings, and improvement recommendations

Job Requirements

• Bachelor’s degree in Computer Science, Data Science, Engineering, or a related discipline (advanced degrees are advantageous). 
• Demonstrated experience with GenAI evaluation tools (e.g., OpenAI Evals, RAGAS, or proprietary scoring systems)
• Familiarity with leading cloud-based AI platforms (e.g., OpenAI, Gemini, Llama)
• Prior involvement in AI quality assurance, ethical AI initiatives, or governance programs
• Strong grasp of Generative AI concepts, including RAG architecture, embeddings, and LLM evaluation techniques.
• Proficiency in query analysis, data validation, and performance benchmarking
• Knowledge of AI evaluation metrics such as accuracy, precision, recall, BLEU, and ROUGE
• Exposure to feedback loop systems and reinforcement learning methodologies is a plus
• Basic programming capabilities (Python preferred) for QA scripting and analytical tasks
• Exceptional attention to detail and analytical thinking 
• Ability to design and implement robust test plans and evaluation frameworks
• Skilled in sentiment analysis and text-based feedback interpretation

Benefits

• Competitive Salary
• BPJS Ketenagakerjaan and Kesehatan
• THR
• External Health Insurance
• Working Tools Provided