We’re putting $5.5M behind research and development of smaller, specialized AI models for real-world deployment — Read our manifesto →
    All posts

    [Case Study] From 7,000 Questions to 100,000 Documents: Morgan Stanley's Eval-Driven RAG

    How a regulated wealth business made institutional knowledge usable, achieved more than 98% advisor-team adoption, and treated evaluation as daily infrastructure.

    Conscious Engines

    Industry: Wealth management
    Organization: Morgan Stanley
    Use case: Advisor knowledge retrieval and meeting support
    Evidence basis: OpenAI customer case using Morgan Stanley-reported results
    Disclosure: This is an independent analysis by Conscious Engines. The source is published by the model provider.

    1. Outcome at a Glance

    Morgan Stanley reports that its assistant reached more than 98% adoption across financial-advisor teams. The knowledge layer expanded access from a previous system oriented around about 7,000 curated questions to approximately 100,000 documents.

    Key Outcomes

    More than 98%

    Advisor-team adoption

    Broad workflow use reported in the cited case.

    About 100,000 documents

    Knowledge corpus

    Enterprise retrieval scale reported in the cited case.

    20% to 80%

    Estimated access to relevant knowledge

    Company-reported improvement.

    MeasureReported resultInterpretation
    Advisor-team adoptionMore than 98%Broad workflow use
    Knowledge corpusAbout 100,000 documentsEnterprise retrieval scale
    Previous access modelAbout 7,000 questionsEarlier curated approach
    Estimated access to relevant knowledge20% to 80%Company-reported improvement
    Quality practiceDaily regression testingContinuous evaluation

    The organization also used speech recognition and GPT-4 to create meeting summaries and draft follow-up content for customer relationship management workflows. Morgan Stanley's Jeff McMillan described the goal as making an advisor “as smart as the smartest person in the organization.”

    Adoption is a strong signal, but it is not a direct financial ROI measure. The public case does not disclose saved hours, revenue lift, error rate, or a complete cost model.

    2. The Operational Problem

    Financial advisors work across research, product material, investment commentary, policy, procedures, and client context. Relevant knowledge exists, but it is distributed across a large controlled document estate. Keyword search requires the user to know the right term and read multiple sources.

    The previous question-based approach could answer known questions but did not expose the full document corpus. Generative retrieval creates broader access, but in a regulated environment an unsupported or outdated answer can create customer and compliance risk.

    The system therefore needed to satisfy four objectives at once: retrieve the right authorized material, generate a usable answer, provide traceable evidence, and behave consistently as models and documents changed.

    3. What Was Built

    The assistant combined retrieval-augmented generation, enterprise documents, access controls, and a rigorous evaluation program.

    System at a Glance

    Ingestion

    Parse and classify controlled research and policy documents.

    Retrieval

    Select relevant passages under the user's permissions.

    Generation

    Produce a concise advisor-facing response.

    Citation

    Link the answer to source documents.

    Evaluation

    Run expert-scored and daily regression tests.

    Meeting workflow

    Use speech recognition and generation for summaries and follow-up.

    LayerFunction
    IngestionParse and classify controlled research and policy documents
    RetrievalSelect relevant passages under the user's permissions
    GenerationProduce a concise advisor-facing response
    CitationLink the answer to source documents
    EvaluationRun expert-scored and daily regression tests
    Meeting workflowUse speech recognition and generation for summaries and follow-up
    Data protectionApply enterprise controls, including zero data retention arrangements

    The daily regression practice is the core lesson. RAG quality can change when the model, prompt, embeddings, chunking, ranking, or source documents change. A frozen set of advisor questions and expected evidence makes that change visible.

    Human experts are required to judge not only factual correctness but suitability for an advisor workflow. The best answer may depend on product, jurisdiction, client type, and current policy.

    4. How It Reached Production

    Morgan Stanley's adoption reflects more than a model choice.

    Curate and permission the corpus. Every document needs ownership, effective dates, product tags, jurisdiction, and access rules.

    Build expert evaluations. Representative questions should test retrieval recall, answer groundedness, citation quality, completeness, refusal, and outdated-content handling.

    Run regression daily. Model and corpus changes should be scored before release, with rollback when a critical task worsens.

    Fit advisor work. Search, summary, translation, meeting preparation, and CRM follow-up should appear in the tools advisors already use.

    Keep client communication controlled. Drafts and meeting summaries require advisor review. High-impact recommendations should remain governed by suitability and supervisory processes.

    The 20% to 80% knowledge-access claim is an internal estimate and needs a defined denominator. Other firms should measure successful retrieval on their own question set instead of copying the percentage.

    5. What Financial-Services Leaders Should Take Away

    Morgan Stanley's case shows that enterprise RAG is a quality system, not a vector database. The valuable assets are the governed corpus, permission model, expert test set, workflow integration, and daily regression process.

    A production-ready private knowledge and voice layer is built around those assets. Task-specific retrieval, reranking, smaller answer models, speech-to-text, deterministic citation checks, and model routing can lower cost while preserving a stronger model for difficult queries.

    The primary metric is cost per advisor-accepted answer with verified evidence, paired with retrieval recall, citation precision, time to answer, and downstream correction. More than 98% adoption shows workflow fit. Continuous evaluation makes that fit maintainable.

    Sources