About Patronus AI

Patronus AI is the leading automated AI evaluation and security company. We are on a mission to boost enterprise confidence in generative AI. Our world-class platform enables enterprise development teams to score LLM performance, generate adversarial test cases, benchmark LLMs, and more. Customers use Patronus AI to detect LLM mistakes at scale and deploy AI products safely and confidently.

Our founding team comes from top applied ML and research backgrounds, including Facebook AI Research (FAIR), Airbnb, Meta Reality Labs, and quant finance. As a team, we have published research papers at top ML conferences (NeurIPS, EMNLP, ACL), designed and launched Airbnb’s first conversational AI assistant, pioneered causal inference at Meta Reality Labs, exited a quant hedge fund backed by Mark Cuban, and scaled 0→1 products at high growth startups. We are backed by Lightspeed Venture Partners and high profile operators like Amjad Masad, Gokul Rajaram, and Fortune 500 executives and board members. We are advised by Douwe Kiela, Adjunct Professor at Stanford University and former Head of Research at HuggingFace.

Responsibilities

As the QA Engineer for Patronus AI, you are responsible for ensuring the quality and reliability of our AI products and systems. You will work closely with our engineering, AI and design teams to identify and resolve any potential issues or bugs, create testing plans and datasets, ensuring that our products meet the highest standards of performance and functionality. Successful applicants are skilled, have prior QA experience and are detail-oriented.

In this role, you will:

• Design, execute, develop and lead QA processes. Taking responsibility for cross-team collaboration, development, planning and extending the team.
• Develop and execute test plans to ensure the quality of software features. Create high quality, specific test cases that cover diverse scenarios and edge cases to catch unexpected behavior.
• Ensure that production releases are well tested. Typically releases happen on a weekly cadence; QA will ensure reliability of new features in the web platform, collaborating closely with frontend engineers
• Regularly report bugs and issues from testing results and file tasks; Triage to engineers accordingly
• Provide feedback to product owner on areas for improvement
• Collaborate with the engineering and AI teams to understand requirements for testing; Participate in technical design meetings as needed
• Construct datasets for AI research, working with the AI team, including datasets for model training, finetuning and evaluation
• Develop and maintain automated testing scripts for our API
• Help customer success team investigate and troubleshoot customer-reported issues to identify root causes and recommend solutions
• Executing load tests
• Run benchmarking scripts to assess evaluator performance from configuration changes, eg. a change in the model selection, underlying dataset or other parameters

Qualifications

“The number one qualification to succeed in this machine learning course is gumption” - John Lafferty, CS Professor at Yale

Above all, we look for a proactive mindset, willingness to learn, relentless drive, and passion for engineering and product. You are a great fit if you have a background in the following:

• Minimum of 3 years of experience as a QA engineer working on web platforms in a production environment
• Minimum of at least 2 year of experience working on test automation with Python or Javascript
• Experience with REST APIs
• Experience performing manual application tests
• Experience writing end-to-end tests
• Experience with Selenium, Cypress or Playwright
• Experience with tools like Postman, Swagger, OpenAPI
• Have good character, integrity, and respect for others!

Nice to haves:

• Advanced proficiency in English
• Experience with SQL
• Experience with CI/CD tools like Jenkins, GitLab, GitHub
• Certification or experience with advance ISTQB or other testing qualifications

Benefits

• Competitive salary and equity packages
• Unlimited PTO
• Fun global offsites!