Selecting a CMS: Managing Product Demos

If you followed the advice from the first two articles in this series (How to build a short listand Developing scenarios), you should have a good idea of what you are looking for and with what products you might find some content management system bliss. This next article provides guidance on how you can start evaluating actual products against your defined requirements.

This next phase of the selection process is where you evaluate the products against your requirements. Successful completion of this phase will mean that you have selected a product/vendor that is compatible with your content and your way of working. The product satisfies both your objective and subjective criteria.

Failure in this phase means that you will either be swayed by the most charismatic salesperson or that you will be stuck in a never ending sales cycle that doesn't drive you towards an informed decision. Neither case is very appealing — so lets avoid both.

Take Product Demos Seriously

The vendor presentation and product demonstration is one of the most critical components in a CMS selection process. You will learn more from seeing a product in action than reading an analyst report or a RFP response from a vendor.

But to be effective, a product demonstration needs considerable investment from both sides. You won't learn anything by occasionally peaking up from your email to glance at a canned demo about a fictional business that has nothing to do with your company.

Instead, you should partner with the vendor to develop a prototype that supports the scenarios you have given in the RFP. In this exercise, you get to experience what it feels like to be a customer working with the vendor to achieve success. If you run a demonstration properly you will be able to answer the following questions:

• Does the vendor understand your business and the way you work?
• Will you be treated like an important customer?
• Does your company and the vendor have good chemistry?
• How naturally does the product fit your vision?
• What customizations or compromises would have to be made to use this product?

Construct a Thoughtful Invitation

The first thing you need to do is connect with the vendors on your short list to tell them that you are evaluating their product and could use their help. This is where most companies go wrong. By pulling out their standard RFP template and loading it up with demands one shoots oneself in the foot, and early.

The worst of such offenses have a glob of irrelevant boilerplate text and then a long feature checklist. One CMS vendor I know even received an RFP that asked if the product contained any radioactive materials — clearly this was language designed for another type of procurement and the customer was too lazy to even read what he sent out.

Trust me, when you do work like this, you are sending a signal to the vendors that you don't care. Some vendors will not engage at all. Others will play along but invest as little as possible in the opportunity — they know that the sales cycle will be long and unpredictable.

You get what you give: an RFP with a lot of boilerplate text will get responses with a lot of boilerplate text. Plus, you will get stuck in "qualification" queue until you show some signs of intelligent life.

The good news is that, if you have done the work of developing scenarios, you have a lot of information that shows (a) you are serious about this initiative and, (b) you know what you want.

Vendors love scenarios because they efficiently tell the story of content in your organization and help them understand what you need. In addition to your 10 most important scenarios, your RFP should contain the following information:

• Background about your company and division
• Sample content types and perhaps some screen captures of how they appear on your site
• A roadmap of your selection process with a timetable
• A point of contact
• The response format you you expect.

Be Concise in Your Communications

Attributed to an impressive number of famous folk including Mark Twain, Pascal and a host of others, there's a quote that I find to be a useful and humorous reminder. It goes something like this: "

If i'd had more time, i'd have writen a much shorter letter."

The most important part to keep in mind is that assembling your RFP is not a contest for who can write the longest, most elaborate proposal. The RFP response will not help you manage your content and the quality of an RFP response says nothing about the product. In fact, the prettiest proposals are usually written by a dedicated proposal writer and re-used across lots of sales opportunities — they have very little to do with your RFP.

Some RFPs demand that participants spend weeks of time filling out a response. You don't want the vendors to spend all of their effort on the RFP and then coast through the rest of the sales process. Vendor resources are much better spent building a prototype that shows you how the product would work in your organization. This will give the vendors a chance to show how they approach problems and how their products work. Keep things as informal as you can.  The more leeway you give, the easier it will be easier to identify differences between the vendors and products. When differences are more visible, decisions are easier.

If you really want a vendor to put in the effort to get to know you and translate their features in terms of your requirements, you should let it be known that you are only evaluating 2-3 products. This indicates that you are in the home-stretch of your decision and it is time to pull out the stops. This will help the vendors justify putting their best people on the opportunity rather than pacing themselves for a long slog. Time-boxing a decision helps people work more efficiently on both sides.

Preparation

A successful demo is all about preparation. You need to prepare the vendor — or systems integrator or in house staff if you are evaluating non-commercial software — with the information they need so they can do their best. You also need to prepare the audience on what they should be looking for.

• Verify that the vendor understands your requirements
  Have the vendor prepare a written response describing how their product can support your scenarios. Review it and give them feedback with ample time to adjust their demo in case they misunderstood what you need. I typically encourage vendors to do a pre-demo walk-through of the scenario in front of their customer contact person. If you are a vendor, always take advantage of this offer. In my own selection consulting work, when one of the three candidates does a pre-demo walk-through, the demos are so much better that they win 100% of the time.

• Prepare the audience 
  Prepare your audience for the demo by telling them what they should be looking for. A scorecard that lists the scenarios is useful for keeping people's attention on their needs, not gimicky features. Vendors tend to up their game when the realize they are dealing with a sophisticated audience.

  If the audience does not understand basic content management theory (separation of content and layout, re-usability, content life-cycle, etc.) address that before the first demo. Vendors are actually pretty good at explaining that stuff but there are more effective uses of their time.

• Call the vendor's references
  Don't wait to the end of the process to call references. If you talk to references before the demonstration, you will be more educated for the demo. Maybe a complaint from a reference was addressed in a newer version of the software. Maybe a feature that demos really nicely isn't practical for everyday use.

Demonstration

The demo should use everyone's time as effectively as possible and should be organized to ensure that vital information is communicated to the right people. I usually allocate 4 hours but the agenda is broken up so that not all stakeholders have to sit through the whole thing.

• Limit company background information 
  The vendor should be able to introduce their company and make the case that it is a stable company, it gets content management, and knows your business. However, you need to contain the amount of time that they take to do it. They should be able to build a level of credibility and comfort with the audience but not infringe on the time they have to talk about their product within your context. Your short-listing exercise should have already pre-qualified the vendors along these lines.
• Mind your manners 
  Even if your corporate culture thinks it is OK for staff to attend meetings in-body only, keep distractions to a minimum. Ask your audience to put aside their email, blackberries, and cell phones and pay attention. Give the vendors every opportunity to engage with the audience. If the vendor is missing the mark, don't tune out. Instead, help steer them back on course. If you can't do that, politely end the meeting as quickly as possible and be happy that you were able to eliminate an option in a very hard decision.
• Mark your scorecards
  Without making it feel like a Bingo hall, have the audience take notes in their scorecards so that they remember what they saw and their impressions. By the time they have gotten back to their desks and answered their first of fifty waiting emails, they will have forgotten half of what they saw. The most important thing for your audience to capture is their doubts. These are aspects of the product or service that raise questions and concerns. The follow up phase will focus on these doubts.

Follow up

Don't let wait long to get feedback from the audience. It doesn't take long for people to forget. Follow up and plan the next steps as soon as possible.

• The post mortem
  As soon as possible, get everyone in a room and have them express their observations and impressions and, most importantly, doubts. For each doubt, you need to first validate (was this a misunderstanding or an oversight?), mitigate (what compromises or customizations could compensate for this issue?), and rate (what risk remains after mitigation? Is the customization expensive or does it risk future upgrades? Is the compromise sustainable?).
• Schedule follow ups
  Review the doubts that came up and have the vendor invalidate or suggest mitigation strategies. For the vendors that didn't make the cut, explain why. If the demo was a disaster but you think the product still has potential, you could give them another chance or you could take it as a sign that they are not prepared to support you. Remember, after the contract is signed, things are only going to get worse.
• Prototype
  Some doubts will be best addressed with a prototype that the vendor can leave behind for you to use. Different vendors will have different policies around this. Some create hosted sand boxes and allow business users to experiment. Others provide trial versions of the software so that a customer can attend training and try to build the prototype themselves.

Bookmarks