A look into the evaluation process for the Forrester Wave™ for CIAM 2022
When considering a technology investment, many organizations seek the expertise of industry analysts, often turning to the most respected firms and popular industry reports as a way to shortlist their vendor selections. From my perspective, as a vendor undergoing the evaluation process, it's been interesting to observe the different approaches researchers take as they analyze companies and their product offerings. In this post, I'd like to explore our experience with one firm in particular, Forrester, in its creation of The Forrester Wave™: Customer Identity and Access Management, Q4 2022.
Everyone (myself included) loves to see how different vendors stack up in a report's final chart or diagram. But have you wondered what goes into those findings? How do the analysts reach their conclusions, and how do they ascertain that vendors can actually do what they claim they do? We feel it's not always obvious in the rankings, nor can you derive a lot of insight from the short vendor descriptions that are a part of most reports.
I was deeply involved in the request for information (RFI) with Forrester for the 2022 report, and I believe I can shed some light on the evaluation process.
Typically, analyst reports begin in similar fashion: the team conducts primary research on a given technology area to develop a list of vendors to consider for evaluation. From that initial list, the vendors are narrowed down based on various inclusion criteria until the final list is determined. While different firms take different approaches to reach their conclusions, this is the point at which Forrester will gather details on the product and strategy of each vendor through a detailed questionnaire, along with use-case demos, briefings, and customer reference interviews. Forrester takes those inputs, along with each of its analyst's experience and expertise in the marketplace, to score vendors using a rating system that compares each vendor against others in the evaluation.
The approach taken by Forrester in its Wave for CIAM report is extensive, much like a full vendor selection, evaluation, and proof process we would see from an enterprise customer and in some cases moreso.
For this report, 15 vendors were asked to provide a thorough RFI response detailing their strategy and market presence. As the head of the sales engineering organization at ForgeRock, I've been involved in our response to numerous requests for proposals from prospective customers, including some of the world's largest enterprises, and I've rarely seen any request as exhaustive as the one submitted by Forrester for this report. Their evaluation included 22 criteria, including detailed descriptions of our product vision, roadmap, market approach, partner ecosystems, delivery models, revenue, and number of live installations, including such details as the largest number of customer authentication attempts per hour at a single client organization. We were also asked to supply a list of customers so that Forrester could conduct reference interviews. Which they did.
Each vendor also had to document how it delivers a set of key use cases. The use cases for the latest Forrester Wave™ for CIAM included:
- Data orchestration: configure intelligent workflow scenarios (aka low-code approaches to deployments). What kinds of visual workflows are available?
- Users and roles: manage role-based access control (RBAC), including adding users to roles and roles to users, editing fine-grained permissions, and using role inheritance and/or embedded roles.
- Customer IDV and registration: configure identity verification of a new user. What out-of-the-box integrations are available?
- Consent management: configure privacy-by-design (data residency) for customer PII that the CIAM solution stores. Show configurations for multiple customer countries, geographical segmentation of users, etc.
- Authentication methods: configure protection against credential stuffing, account takeover, and password spraying, and how to enable single sign-on (SSO), passwordless, biometrics, and tokens for multi-factor authentication.
- Risk-based authentication: configure risk-based customer authentications and rule-based risk scores, and configure which authentication methods will be invoked based on the risk score.
- Customer-self-service: configure policies for customers to recover a forgotten user ID, manage their devices, update their profile, and more.
- Business systems integration: configure the CIAM solution to integrate with CRM (e.g., Salesforce), MDM, web analytics, ecommerce portal, and others. What features are available beyond SCIM?
- IDV and fraud management: configure the CIAM solution to integrate with third-party IDV solutions, such as Equifax, Experian, LexisNexis, TransUnion, etc., and with an Enterprise or Retail Fraud Management solution, among others.
- Reporting, dashboarding, and scalability: set up, define, and run an ad-hoc report. Show filtering, changing order of columns, hiding/showing columns in the output. And show how to set up customized dashboards for CIAM administrators.
And this is where it is a little different with Forrester. In addition to written descriptions, they required vendors to demonstrate each use-case scenario in real time to the analyst, which typically took three hours. Under each of the use cases listed above were five or more specific use cases. Vendors had to document and demonstrate their effectiveness in all of them, for a total of 50 use case demos. Beyond just responding to and demonstrating the use cases documented, the analyst can "go off-script," asking the team to demonstrate other capabilities or prove something out a little further — this is a real examination of both the platform and the team presenting.
The use cases Forrester examined for its analysis represent a comprehensive set of requirements for any modern CIAM solution. We believe that any organization considering buying and implementing identity for their customer- or citizen-facing services should be looking at these capabilities and understanding why certain vendors outperformed others.
The team at Forrester has many years of experience in the IAM and CIAM segment and rigorously tests vendors. But we believe this rigor is exactly why ForgeRock obtained the highest score possible in 15 criteria, including data orchestration, users and roles, customer IDV and registration, consent management, authentication methods, risk-based authentication, and customer self-service. And ForgeRock earned the highest score amongst all vendors in the current offering and strategy categories.
When investing in an essential technology, like CIAM, it's important to examine the capabilities that matter most to your organization's success. Reading The Forrester Wave™: Customer Identity and Access Management, Q4 2022 is a great place to start.
