Join to apply for the Lead AI Trainer / Prompt Evaluator role at Social Discovery Group.

Social Discovery Group (SDG) is the third largest social discovery company in the world, uniting 60+ brands with 500 million users. SDG addresses loneliness and isolation by transforming virtual intimacy into a new normal. Our portfolio includes Dating.com, DateMyAge, Cupid Media, Dil Mil, Kiseki, and others. SDG also invests in IT startups such as OpenAI, Patreon, Flo, Clubhouse, Woebot, Flure, Astry, Coursera, Academia.edu, and many more.

Company Overview

Our international team of 1,200 professionals and digital nomads works all over the world, offering remote opportunities from locations including Cyprus, Malta, the USA, Armenia, Georgia, Kazakhstan, Montenegro, Poland, Latvia, Serbia, Spain, Portugal, UAE, Israel, Turkey, Thailand, Indonesia, Japan, Hong Kong, Australia, and many others.

In August 2024, we achieved Great Place to Work US Certification™.

Responsibilities

• Review and assess the quality of datasets produced by the content team, ensuring accuracy, consistency, and adherence to labeling guidelines.
• Provide constructive feedback and help maintain a high quality bar across annotators.
• Personally perform high-quality annotation and dataset creation as a subject‑matter expert.
• Validate conversational AI model outputs (e.g., SbS evaluations, error tagging, scenario stress‑testing) to detect issues before production release.
• Work closely with the ML team to refine instructions, dataset specifications, and evaluation criteria for model training cycles.
• Identify recurring issues in annotations or model behavior and suggest improvements to workflows, QA processes, or validation pipelines.
• Contribute insights on prompts and data strategies to accelerate model quality improvements.

Qualifications

• Experience in dataset annotation and content validation for conversational AI / chat models / NLP products.
• Hands‑on experience with SbS (side‑by‑side) or other model evaluation frameworks (e.g., quality scoring, tagging, error categorization).
• Ability to analyze conversational responses and identify accuracy, tone, safety, and contextual relevance issues.
• Experience preparing or following annotation guidelines and QA checklists.
• Strong attention to detail and ability to consistently apply evaluation criteria.
• Ability to provide clear and constructive written feedback to other annotators.
• Familiarity with prompt engineering practices is a plus.
• Familiarity with LLM‑based automated evaluators / AI‑as‑a‑judge is a plus.
• English proficiency (B2+) in reading and writing.

Benefits

• Remote opportunity to work full time.
• Vacation: 28 calendar days per year.
• 7 wellness days per year (time off for household issues, rest without sick leave).
• Bonuses up to $5,000 for recommending successful applicants.
• 50% payment for professional training, international conferences, and meetings.
• Corporate discount for English lessons.
• Health benefits: up to $1,000 gross per year per employee if not eligible for corporate medical insurance; can be used for self‑purchase of health insurance or doctor’s fees for you and close relatives.
• Workplace organization: company provides an equipped workplace and all necessary equipment (table, armchair, wifi, etc.) in offices or co‑working locations; reimbursement of workplace costs up to $1,000 gross every 3 years for other locations.
• Internal gamified gratitude system: receive bonuses from colleagues and exchange them for merchandise, team building activities, massage certificates, etc.

Job Details

• Seniority level: Mid‑Senior level
• Employment type: Full‑time
• Job function: Education and Training
• Industry: Software Development
• Location: Warsaw, Mazowieckie, Poland

Apply

Sounds good? Join us now!