Case Studyagentsamazon bedrockevaluation frameworkchurch management
Pushpay Builds Agentic AI Search For Ministries
8.3
Pushpay developed an agentic AI search for church administrators using Amazon Bedrock and Claude Sonnet 4.5, enabling natural-language queries against community data. The team implemented a generative AI evaluation framework with a curated golden dataset of over 300 queries and instrumentation to improve accuracy beyond an initial 60–70% success rate, reducing time-to-insights from minutes to seconds for early users.
Key Points
- 1Builds agentic AI search using Amazon Bedrock and Claude Sonnet 4.5 LLM
- 2Addresses slow insights with a generative evaluation framework and golden dataset of 300+ queries
- 3Enables product teams to measure accuracy by domain and iterate to production faster
Scoring Rationale
Practical architecture and evaluation approach drives usefulness, but limited novelty and vertical focus constrain broader impact.
Sources
Public references used for this report.
Practice with real Retail & eCommerce data
90 SQL & Python problems · 15 industry datasets
Used by DS/ML engineers at top companies
250 free problems · No credit card
See all Retail & eCommerce problems
