AI Evaluation Specialist

Responsibilities:

• Design and implement robust automated evaluation frameworks (using Python) test LLMs for tasks like reasoning, coding, and summarization.
• Lead the development of annotation rubrics and manage workflows for human evaluators to generate high context preference data and golden datasets.
• Design and execute adversarial testing (re-teaming) to identify vulnerable, hallucinations, and biases in mode outputs before deployment.
• Develop and calibrate reliable LLM-based evaluators to replace human raters at scale for specific metrics, validating their correlation with human judgment.
• Analyze evaluation results to pinpoint specific model weaknesses (e.g. model fails at multi- step reasoning in finance contexts) and present actionable insights to modeling and product teams.
• Build and maintain internal evaluation in the platform and dashboards to track model performance across different versions and use cases.

Requirements:

• 4+ years of experience in machine learning, Data science, or AI Evaluation
• A degree in Computer Science, Information Technology, Data Science, or a related field.
• Proven track record of designing evaluation strategies for NLP or Generative AI products.
• Expert-level proficiency in Python for scripting evaluations and analyzing results (pandas, NumPy).
• Strong ability to query data (SQL) and perform statistical analysis to validate evaluation confidence intervals and inter-annotator agreement.
• Advanced ability to craft prompts for crafts prompts for both model testing and steering LLM-based evaluators.

Remuneration: NGN 500,000