Senior MLOps Software Engineer

Your daily tasks: - Design, build, and own the software solution used to assess cyber capabilities in AI models. - Define a measurable benchmark and evaluation methodology covering relevant cyber-capability scenarios, tasks, scoring criteria, thresholds, and controls. - Implement repeatable, automated evaluation pipelines that execute benchmark tests and generate consistent model-level results. - Analyze model outputs, quantify performance, investigate failure modes, and translate findings into clear technical evidence. - Partner with Model Risk Management, cyber security, data science, engineering, and governance stakeholders to align the approach with review expectations. - Document evaluation assumptions, scope, data sources, test cases, metrics, limitations, versioning, and decision rationale to support independent challenge and auditability. - Build reliable software components, APIs, data-processing workflows, and reporting outputs using secure engineering practices. - Integrate evaluations into CI/CD and MLOps workflows, including experiment tracking, reproducibility, monitoring, and controlled release processes. - Maintain benchmark relevance as model capabilities, cyber risks, and evaluation techniques evolve. - Provide technical leadership, conduct code and design reviews, and drive the solution from proof of concept through production ownership. Your attitudes we value - 7+ years of professional experience in software engineering, MLOps, machine learning engineering, data engineering, or a closely related discipline. - Strong software engineering skills, preferably with Python, including modular design, testing, version control, APIs, and production-quality coding standards. - Hands-on experience evaluating machine learning or generative AI model performance using quantitative and qualitative methods. - Strong data analysis skills, including metric design, experiment analysis, error analysis, statistical reasoning, and clear visualization or reporting of results. - Practical experience developing automated model evaluation or benchmarking frameworks and converting evaluation outcomes into decision-ready evidence. - Experience interfacing with Model Risk Management, independent validation, risk, compliance, or governance teams. - Working knowledge of MLOps practices such as CI/CD, model and dataset versioning, experiment tracking, reproducible execution, monitoring, and deployment controls. - Ability to own a technical solution end to end, from requirements and architecture through implementation, validation, documentation, deployment, and support. - Strong written and verbal communication skills, with the ability to explain complex evaluation results to technical, risk, and non-technical stakeholders. What we offer: - Hybrid working model (2 days in the office from our Warsaw office, 3 days remote per week) - Competitive salary with annual salary reviews - Opportunity to use foreign languages and the newest technologies on a daily basis - Indefinite period employment contract after probation period - Access to MyBenefit platform where you can choose from a variety of benefits (Sport, Tourism, Culture, Recreation etc.) - Sport's card (Multisport) - Private medical care for you and your family - Private life insurance - Access to wide range of learning & development platforms - Discounted theatre tickets - Work in a multicultural environment