Software Evaluation

How long has this been live?

• You will often get a range (like 5-7 years) or a "don't know" to this question. We can accept a single number, so ask them what number they feel comfortable with or ask them for at least the minimum amount of years. If live >2 years try to collect a DI. They must have DECIDED within the last two not just been live that long.

If known, What version of the product are you on?

• If they don’t know if they’re current: Do you get regular updates?

• Sometimes the only people who know this answer are in IT. The version # may be listed on the login screen. Sometimes providers want to hunt to try and find it and that can bring the conversation to a grinding halt. Try to redirect them by asking them if they get regular updates. If so, type in "Current". If they don't know, don't leave it blank. Type in "Unknown". We are just trying to figure out if they are using a product that is in general use, so we're comparing apples to apples or if they're using a product that is out of date and general use.

Product is part of your long-term plans

You plan to keep this solution for the next 3-5 years

•  If No: What is driving you organization to look at other solutions?

• If the answer is no, ask if they will be deciding on a new solution within the next 2 years. If so, collect a DI.

Would you buy this again?

Knowing what you know now and the competitors that exist today, would you buy this solution again, if you had to make a choice?

Keeps ALL promises

For example, the vendor delivers expected functionality, keeps development timelines, implements on time, and resolves concerns they promised to address in a reasonable timeframe.

• Always get commentary!

• This question is occasionally left blank because it is a very BIG question and they don't feel comfortable picking one or the other.  (Usually, because they kept most of their promises) If that's the case, give them the opportunity to score it or leave it blank. Always get commentary! If Yes, get examples or praise. If No, get examples and ask if there were any promises that the vendor DID keep.

Avoids charging for every little thing

The vendor avoids charging unexpected costs that you feel are unfair, too frequent, or too high.

• This question is worded in a way where yes results in a positive score and no is a negative score.

• Not everyone has visibility here. Give the provider the option to score it or leave it blank. This question has tripped a lot of providers (and researchers) up.

• Click and drag your mouse to highlight the word "Avoids" so the colors are inverted from the rest of the sentence.

• Put vocal emphasis on the word Avoids as you read it. 

• If there is still confusion, restate the question by saying “Yes the vendor avoids charging for every little thing, or no, the vendor does not avoid charging for every little thing.”

Quality and effectiveness of implementation services, including recent upgrades

How well the vendor supports you in the go-live process or significant upgrade to this solution.

• If they've had the product for a long time, we want to find out about their most recent upgrades. The frequency for upgrades varies widely for vendors and organizations. Some products get upgrades multiple times a year, and some only every year or so. We want to stress that it's the latest upgrade(s) in the last one or two years. If it has been a while, let them know they can leave the question blank because we are only measuring recent experiences.

Quality and effectiveness of Initial and ongoing training

How effectively the vendor trains you on functionality, workflow best-practices, use cases, etc.

• Training can take many forms and we are interested in rating anything that applies.

• Training can be in-person, video conferences, online tools, tutorials, webinars, chat features, printed materials, etc. It doesn't have to be tied to an implementation, it can be offered on a schedule, or available as needed.

• If it is low scoring, ask how they would design a training program for their organization. This is a BIG issue, as there are many different aptitude levels for healthcare workers. Technology can be very intimidating for some. Provider satisfaction depends heavily on the quality of training they receive. (Just ask anyone on the Arch Collaborative team what they have to say about that!) By having the provider detail the content, method, and frequency they want, we're able to provide a roadmap for the vendor. 

Product works as promoted

The solution meets the expectations as set by the vendor during the sales process and/or upgrade discussions.

• Sometimes providers are hesitant to answer this question because they weren't in the sales/upgrade meetings. They don't have to be in the meeting with the vendor to be able to answer this question. We want to know if it meets general expectations

Product supports your organization’s integration (interoperability) goals

How well this solution exchanges needed information in a helpful/usable way with other internal and external solutions.

• Providers don't always understand what this is asking, so read the definition right after you ask the question. It's possible that it's a standalone product and they don't need it to integrate. If that's the case, give them the opportunity to score it or leave it blank. 

Overall product quality, including upgrades

Design, stability, reliability, and/or responsiveness of this solution

• This question is pretty straightforward. There are multiple attributes that go into this question so try to find out which ones are feeding into their score.

Ease of use for end users

The solution is simple and intuitive when it comes to training new users, navigation, and the time to complete tasks

• Occasionally providers will ask if we're asking if it's easy for them or for everyone. Give the provider the freedom to answer this however they like. This evaluation is ultimately measuring their experience but they may have insight on shared experiences with others who have a different experience with the solution.

• And end user is anyone who uses the product.

Current product functionality meets all your needs today

As it exists today, this solution offers everything you need to achieve your current goals and outcomes.

• This is a great question for getting innovative "wish list" ideas from providers.  Try the phrase, "End users often drive innovation" before asking for a wish list. This helps them understand we view them as an agent of change. Many end users don't have the power to make decisions about buying software. They may feel powerless when working with the tools they have been given to do their job. This question gives them an opportunity to help improve a tool they have to use. This is very empowering to them! Take time to explain this concept to them. It will result in great discussions, an overall positive call experience for the provider, and a roadmap for the vendor to develop their product. 

Delivers new technology that meets your needs

This vendor has a vision on where the solution needs to go and consistently delivers on that vision.

• If the product is new they may not have had time to see the evolution of the product or they haven't seen much change so they don't feel like they can answer. If that's the case, give them the opportunity to score it or leave it blank.

Quality of ongoing support (phone/web)

The vendor helps you fully resolve problems in a timely manner.

• Sometimes providers get hung up on the phrase "phone/web". It can be any avenue. It can help to get a description of the process.

• Try reading the definition right after you read the question to avoid confusion about what that means.

Service is proactive

The vendor takes the lead in alerting you about potential problems, provides guidance and insights, and initiates helpful communication. Anything that is the opposite of reactive.

• This is a great actionable insight question for the vendor. We want to know 3 things (what, when, and how). What content do they want to hear about (i.e., Technical issues, software updates, roadmap plans, Q&A discussions)? When do they want to hear from them (i.e., How often is often enough or what is too much)? How do they want to interact with them (i.e., Do they want e-mails, phone calls, virtual or in-person meetings?) Find out what partnership looks like to them. A lot of providers only hear from the vendor when they have a new pitch to sell them on something, so look out for that.

Vendor Executive involvement

You work with people at the vendor – at any level of the organization – who are empowered to solve your problems.

• The essence of this question is about access to empowered people. It doesn't matter if it's their front-line account rep or if they can escalate it, if needed. We want to find out if there's red tape or if they feel like it's smooth.

• Take the following 2 steps to eliminate any confusion:
   (1) Emphasize the words at  any level and empowered.  (2) Raise and lower your hand like you're pointing out the job level in a hierarchy chart to illustrate that it's all in consideration.

Received your money’s worth

The return you receive from the product is worth the investment you put into it.

• What is the major value driver with this solution?

• They don't have to know the exact cost or pricing structure. It can be a general idea of getting a return on investment. If they don't have visibility on the pricing, give them the opportunity to score it or leave it blank. It's not necessarily about the sticker price. Some providers want the most affordable product and some  are willing to pay more for something if they get what they need out of it.  There are a lot of ways to improve satisfaction here. Find out what needs to happen to get to a 9.

Likely to recommend to a peer or friend

Based upon your experience, this solution would be a good fit for a peer at an organization like yours.

• This is a great opportunity to get some advice for their peers. You can explain that this is one of the main reasons people come to our site and they will have the same opportunity to use that information for any product they are considering. By explaining this, you are giving a great plug to use our site and explain that this is one of our main purposes. We want to help them! We want them to know that by speaking to us, they are part of a community with their peers. They can all benefit from each other's wisdom and experience. They all want to see what would be the best fit for their organization and want to see their software improve.

Overall satisfaction

Your overall satisfaction with this solution, considering all your experience with the vendor.

Forecasted overall satisfaction in 12 months.

Your predicted overall satisfaction next year, based upon current trend

• If there is a change in the score (up or down) ALWAYS get commentary about why they think it will change.

Drives tangible outcomes

This solution reduces pain points and helps you achieve expected, measurable outcomes.

• Always read the full definition.

• What outcomes/benefits have you seen as a result of using this software?