Senior Manager, LLMOps

As Senior Manager, LLMOps & AI Integration Engineer (Individual Contributor), you will lead the model integration, agent/tooling, and LLM operations strategy for our pip‑installable AI Capabilities SDK. You will design and maintain clean, extensible interfaces for LLMs, embeddings, retrieval, tool execution, and agentic workflows; build runtime guards and evaluation hooks; and optimize latency, reliability, and cost—so teams can embed AI safely and predictably.

You will collaborate closely with:

• Embedded AI Architecture and Solution Design to prioritize capabilities and define interfaces aligned to real-world solution patterns.

• AI Engineering to deliver PI increments, integrate external technologies, and codify model/agent failure modes and mitigations.

• Digital Creation Centers and Forward Impact Teams (FITs) as consumers of the SDK, capturing feedback to refine ergonomics and performance.

• DevOps, Evaluation & QA, and Developer Experience engineers to wire guardrails, evaluation, and developer-ready assets into CI/CD and docs.

Why this Role Matters

LLMOps sits at the intersection of capability, safety, and operability. By turning complex model/agent behaviors into well-abstracted, evaluable, and observable SDK modules, you enable teams to deliver AI features fast with safety—reducing bespoke implementations, ensuring reproducibility, and controlling latency/cost at scale. Your work is foundational for platform services that rely on the SDK as their backbone.

1) Model & Agent Integration Architecture

• Design Python SDK interfaces for LLM clients, embeddings, tokenization, and structured outputs (e.g., Pydantic/JSON schemas).

• Implement function/tool calling abstractions, agent orchestration patterns (ReAct-like, planner/executor), and MCP adaptors for interoperable tools.

• Provide configuration and secrets integration points that downstream services can adopt consistently (env vars, config schemas, key management hooks).

2) Retrieval & RAG Components

• Build RAG modules (document loaders, chunking/segmentation, embeddings, retrievers, rerankers) with pluggable backends (vector stores, search indices).

• Standardize connectors and interface contracts for common data sources; ensure reproducible pipelines and SDKservice parity of behaviors.

• Optimize retrieval quality via evaluation hooks (precision/recall, MRR, hit rate) and tunable parameters (top‑k, thresholds).

3) Guardrails, Safety & Policy Integration

• Codify safety guardrails: prompt hygiene, jailbreak resistance, content filters, sensitive topic/policy checks, red‑team/adversarial probes.

• Integrate privacy/security controls: data minimization, PII detection flags, safe logging, and deterministic truncation strategies.

• Provide policy‑as‑code hooks and pre/post‑processing middleware to enforce runtime constraints within SDK flows.

4) Performance, Reliability & Cost Engineering

• Implement caching, batching, retry/backoff, circuit breakers, and fall‑back strategies across model/tool calls.

• Profile and tune latency, throughput, and token usage/cost; expose configuration knobs and budget guards (token and cost caps).

• Maintain benchmark suites and regression baselines for model and retrieval components; block releases on significant degradations.

5) Observability‑by‑Design for Libraries

• Embed telemetry hooks (structured logs, metrics, traces) suitable for a library so that services consuming the SDK can attach enterprise observability.

• Define SLIs relevant to LLMOps (latency, error rates, cache hit ratio, token usage, cost per call) and document runbook guidance for downstream teams.

• Standardize error models and return types to ensure predictable handling across consuming services.

6) Evaluation & CI/CD Integration

• Partner with Evaluation & QA Engineers to build test harnesses (unit/integration/contract/fuzz/adversarial) and benchmark suites for agents/tools, RAG, and model endpoints.

• Work with DevOps Engineers to wire quality gates into CI/CD: coverage thresholds, performance budgets, safety checks, SBOM/signing for release artifacts.

• Provide fixtures and simulators to enable deterministic tests across backends (vector stores, providers).

7) Developer Experience & Enablement

• Collaborate with Developer Experience and Technical Writers to deliver quickstarts, sample apps, notebooks, and code recipes for common patterns (RAG, agents, tool use, structured outputs).

• Contribute to cookiecutters/CLI scaffolds that prewire configuration, tests, telemetry, and guardrails for new modules or integrations.

• Support community contribution workflows (feature branches, PR checks, doc acceptance criteria) and mentor contributors on best practices.

8) Collaboration & Continuous Improvement

• Close feedback loops with Embedded Architecture, Solution Design, AI Engineering, Creation Centers, and FITs to refine APIs, defaults, and ergonomics.

• Participate in post‑release retrospectives and incident reviews to strengthen guardrails, performance tuning, and developer experience.

• Track and improve LLMOps KPIs (latency, cost per call, error rates, cache hit ratios, evaluation pass rates).

MEASURES OF SUCCESS

• Performance & cost: Latency and token/cost budgets consistently met; cache hit ratios improved; fewer performance regressions.

• Safety & reliability: Guardrail tests pass; reduced policy violations and incident rates; predictable error handling.

• Adoption & DX: Faster time‑to‑first‑success for consuming teams; positive developer feedback on APIs and examples.

• Evaluation coverage: Benchmarks and adversarial tests integrated; high pass rates; clear baselines with trend reporting.

• SDKConsumer parity: Consistent behavior across downstream services; reduced bespoke integrations and rework.

Basic Qualifications

• 7+ years in software/AI engineering with significant experience in LLMOps or model/tool integration for Python libraries.
• Deep proficiency in Python and building extensible SDKs (interfaces, packaging, semantic versioning, backward compatibility).
• Hands‑on with GenAI/LLM ecosystems (e.g., tool/function calling, structured outputs, prompt orchestration) and RAG components (embeddings, retrievers, rerankers, vector stores).
• Strong grasp of performance engineering (profiling, caching, batching), resilience patterns (retry/backoff, circuit breakers), and cost control (token budgeting).
• Experience embedding observability hooks and integrating evaluation/benchmarks into CI/CD pipelines.

Preferred Qualifications

• Familiarity with agent frameworks, MCP adaptors, and multi‑provider model client abstractions.

• Experience with adversarial/safety testing, privacy/PII considerations, and policy‑as‑code guardrails.

• Background in open-source workflows (feature branches, PR reviews) and multi‑backend integration (vector DBs, search, storage).

• Exposure to regulated environments (healthcare/pharma/finance) and audit‑ready evidence capture.

• Comfortable collaborating across architecture, DevOps, QA, DX, platform, and product teams in matrixed organizations.

Work Location Assignment: Hybrid

Purpose 

Breakthroughs that change patients' lives... At Pfizer we are a patient centric company, guided by our four values: courage, joy, equity and excellence. Our breakthrough culture lends itself to our dedication to transforming millions of lives.  

Digital Transformation Strategy

One bold way we are achieving our purpose is through our company wide digital transformation strategy. We are leading the way in adopting new data, modelling and automated solutions to further digitize and accelerate drug discovery and development with the aim of enhancing health outcomes and the patient experience.

Flexibility  

We aim to create a trusting, flexible workplace culture which encourages employees to achieve work life harmony, attracts talent and enables everyone to be their best working self. Let's start the conversation  

Equal Employment Opportunity 

We believe that a diverse and inclusive workforce is crucial to building a successful business. As an employer, Pfizer is committed to celebrating this, in all its forms – allowing for us to be as diverse as the patients and communities we serve. Together, we continue to build a culture that encourages, supports and empowers our employees.

Disability Inclusion

Our mission is unleashing the power of all our people and we are proud to be a disability inclusive employer, ensuring equal employment opportunities for all candidates. We encourage you to put your best self forward with the knowledge and trust that we will make any reasonable adjustments to support your application and future career. Your journey with Pfizer starts here