Experience: 5+ years
Job Type: Full-time (Contractual / Freelancing)
Location: Remote
In healthcare-related scenarios, accuracy and clarity are essential. This project focuses on evaluating and improving how conversational AI systems respond to medical and healthcare topics. Your expertise helps ensure responses are factually correct, clearly explained, and aligned with real-world healthcare knowledge and communication standards.
What You’ll Do:
- Write and refine prompts to guide model behavior in healthcare scenarios
- Evaluate LLM-generated responses to healthcare-related queries for accuracy, reasoning, clarity, and completeness
- Conduct fact-checking of all medical and healthcare claims using trusted public sources and authoritative references
- Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies
- Assess tone, completeness, and appropriateness of responses for real-world healthcare use
- Ensure model responses align with expected conversational behavior and system guidelines
- Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines
Who You Are:
- You have a minimum of 5 years of real-world professional experience in Healthcare, supported by an associated expert degree (e.g., MD, DO, RN, NP, PA, PharmD, MPH, or equivalent)
You have experience in one or more of the following sub-domains:
- General Clinical Care
- Specialty Medicine or Surgery
- Diagnostics, Imaging & Laboratory Medicine
- Public Health, Healthcare Systems & Administration
- You have significant experience using large language models (LLMs) and understand how and why people use them
- You have excellent writing communication skills for complex medical topics
- You have strong attention to detail and are comfortable evaluating clinical reasoning and medical explanations, identifying subtle inaccuracies or gaps that others may overlook
Nice-to-Have Specialties:
- Prior experience with RLHF, model evaluation, or data annotation work
- Experience writing or editing high-quality medical or healthcare-related content
- Experience in clinical documentation, charting, or patient communication, including explaining medical information to non-clinical audiences
- Familiarity with evaluation rubrics, benchmarks, or quality scoring systems