Problem Statement

Bristol Myers Squibb introduced a new ServiceNow Virtual Agent within the myBMS employee portal to streamline IT and HR ticketing processes. However, because user data was anonymized, stakeholders had limited visibility into how employees interacted with the tool. It was unclear whether users understood service category labels, experienced smooth platform integration, or found knowledge summaries helpful. This lack of insight created uncertainty around the tool’s usability and effectiveness in meeting employee needs.

Team Structure

• IT Stakeholder: Kenneth Meisch

• HR Stakeholder: Michelle Nardo

• HR Liaison/Research Assistant: Julie Prianos

• UX Researcher & UX Writer: Eileen Calub

Timeline

April to May 2025

1. Gather Project Requirements: 1 week

2. Participant Recruitment: 2 weeks

3. User Interviews: 2 weeks

4. Analysis: 1 week

Methodology

• Conducted stakeholder interview with IT and HR stakeholders to identify key focus areas and project goals

• Recruited 10 participants across Enabling Functions with a range of experiences with the Virtual Agent

• Scheduled and moderated 30-minute user interviews on Microsoft Teams

• Conducted thematic analysis of data to identify top user pain points and actionable insights

• Presented improvement recommendations to stakeholders, designers, and developers

• Recognized research participants with compensation in the form of Bravo points, a virtual awards platform at BMS

Project Goals

• Understand how users interact with the Virtual Agent in the context of IT and HR support

• Determine if users are aware of the PTO check feature and whether users prefer to complete transactions within the Virtual Agent interface, as opposed to Workday, Concur, ADP, etc.

• Evaluate the clarity and intuitiveness of the IT and HR service categories presented

• Assess whether users can easily recognize and trust the source links provided in the Knowledge Summary

HR Services categories

IT Services categories

Expanded HR Service categories

Knowledge Summary with Source Links as numbered circles

Findings

• Analyzed 10 user interview recordings total

• Identified 15 actionable insights:

  1. VA Awareness: The majority (90%) of users became aware of the VA because it appears in the bottom-right hand corner of myBMS.

  2. Ticket Submission: Several (40%) users had used the VA for IT tickets, but only one user had submitted an HR ticket.

  3. Value in Providing Quick Resolutions: Most (80%) users say the VA is best at providing “quick answers” and “narrowing down” options.

  4. Seeking More Personalized Answers: More than half (60%) of users felt answers were too “generic.”

  5. PTO Feature Awareness: None (0%) of the users were aware they could check PTO balance through the VA.

  6. Integration Hesitation: More than half (60%) of users expressed reservations about transacting in VA rather than Workday, Concur, and other platforms.

  7. Request Catalog: The majority (80%) of users found the term “Request Catalog” unclear and disconnected from their expectations, suggesting a need for more intuitive or descriptive labeling.

  8. Performance@BMS: Users found the label “Performance @ BMS” confusing and unclear, indicating a need for more descriptive or familiar terminology to improve comprehension.

  9. HR Rebranding: Users found the rebranding of “HR” to “People Services” occasionally awkward, particularly in phrases like “Create a People Ticket,” highlighting the need to test new terminology for clarity and natural language fit.

  10. Global Mobility: Users were unclear about the meaning of “Global Mobility,” associating it vaguely with relocation or immigration, suggesting a need for clearer labeling or contextual explanation.

  11. Employee Data Management: Users made uncertain assumptions about “Employee Data Management,” indicating confusion between system settings and broader employee data, highlighting the need for more precise terminology.

  12. Reporting: Users were unsure whether “Reporting” referred to data metrics or managerial hierarchy, revealing ambiguity that could be resolved with more specific labeling.

  13. Distinguishing Between Categories: Users struggled to differentiate between similar categories like “Payroll” and “Compensation,” suggesting a need for clearer distinctions or supporting descriptions.

  14. Knowledge Summary Quality: Users expected location-specific responses and found irrelevant content (e.g., Hyderabad-based info) unhelpful, suggesting the need for personalization based on user profile data.

  15. Source Link Recognition: Most (70%) users did not recognize numbered references as clickable links, indicating a need for clearer visual cues such as blue text, underlining, or list formatting to improve discoverability.

Thematic analysis of user interview data in Miro

Recommendations

• Provided 3 primary change recommendations to stakeholders, designers, and developers:

  1. Clarify IT/HR Services Wording

     • Reword titles that cause the most confusion:

       1. Request Catalog ➡️ Request Application Access & Device Peripherals

       2. Global Mobility ➡️

          • International Talent Deployment

          • Global Talent Deployment

          • Global Talent Relocation

       3. Employee Data Management ➡️ Requires further clarification by HR prior to renaming

     • Ensure category distinctions are clear

     • Provide hover-over explanations for ambiguous categories.

  2. Provide Source Links Only in Dropdown

     • Remove numbers within the knowledge summary and provide KB links only in the Sources dropdown

       1. Enhances summary readability

       2. Reduces confusion

  3. Consider Privacy Concerns in Integration

     • Ensure that the VA can provide accurate and secure access to personal data

     • Promote the VA as a complementary tool rather than a replacement for existing systems

Findings were delivered to 40 stakeholders and sparked in-depth discussions on IT and HR Services wording beyond the Virtual Agent, including the myBMS portal and ServiceNow knowledge articles.