Oct 18 2009

It’s Not Always The Report Writers Fault

Categories: General, Rants Dave Rathbun @ 11:08 am

I recently attended the 2009 TechEd conference. Here is a copy of the top portion of the session evaluation form. Notice anything that might cause a problem?

I know that form was reduced to fit the size of the page so it may not be immediately obvious what I think the issue is. In order to make it more clear I have posted a larger view of the specific survey choices.

Look at it for a bit and when you’re ready …

I spent several days filling out this evaluation form before I decided that I needed to read it closely. Unfortunately, here’s how I filled them out:

Now do you see the problem? ;) And how it’s not the report writers fault? Once the data is collected, there’s no way to go back and redo it.

Still don’t see what my concern is?

The survey form is not consistent. In four of the five questions the best possible response is all the way to the right. However, for one question (the fourth one) that is not the case. The question in this case was related to how technical the content was. The best answer would be 3 or “Just right” instead of 5 for this one question. By the time I realized that, it was too late. I would love to see the overall speaker survey results, because I suspect I am not the only one making this mistake. Once I saw the pattern from the first few questions on the form, I made the (incorrect) assumption that the rest of the questions followed the same pattern. As a result of my error, the data from my survey submissions are wrong.

Because the form is (in my opinion) poorly designed, the data is likely to be wrong. If the data is wrong, the report will be wrong. Because the report is wrong, the speaker will be given the wrong indication as to the quality of their presentation. And it’s not the report writers fault.

3 Responses to “It’s Not Always The Report Writers Fault”

  1. Comment by Sandor

    I noticed it the minute you hinted at something being wrong with the form. This is actually one of the most common pitfalls when designing questionnaires and is almost always an example in marketing courses that deal with questionnaires and how to design them.

  2. Comment by Mohammed Hussain

    Hi Dave,

    I totally agree with you and nicely observed, this reminds me of you article about IPad’s key pad.
    Moral of the story: Don’t just blindly give someone full marks.

    Mohammed Hussain

  3. Comment by Dave Rathbun

    The survey form for Annual Conference (2011) has a similar problem. For the first few questions the “high mark” is to the left. For the last question they switched, and the left-most mark is the worst result, with the best being the furthest to the right.

    Be consistent, people, please.

Leave a Reply

If you want to include formulas or code in your comment, please read my Tips for formatting comments first. Tags you can use are listed below.

XHTML: You can use these tags: <a href="" title=""> <acronym title=""> <blockquote cite=""> <code> <em> <strike> <strong> <sup> <sub> <u>

Confirm submission by clicking only the marked checkbox:


Please remember that comments that are not related to this blog post may be ignored or deleted without notice. If you're looking for help on a topic you have already posted on BOB then please do not repost your question here.