I had some nice and long story explaining everything I wanted to see. But I
accidentally deleted it. So now I make it short.
First of all. Your questionnaire is not that bad as long as it is not seen as
representing anything meaningful. It can be a starting point for another
questionnaire which goes more deeper into the matter. In the beginning you do
not have to be very extensive. However some things can already be predicted
beforehand and additional questions or adjustments may already be very helpful.
- Proper research question and presentation of the questionnaire. -
You must be aware of the fact that certain quantities are not very well
suitable to be measured by means of a questionnaire and also that certain
questions are not suitable to represent a certain quantity. You make it
yourself very difficult by trying to determine the influence of drinks by
means of a questionnaire and by asking people: 'Just tick off your favorites".
People may have very different ideas about the most influential drinks. It
would be better to guide people a little bit more. Ask more different types of
questions. You should also not want to grasp influence by a single quantity. I
would ask
1) "Which drinks do you feel are currently having the most influence"
2) "Which drinks do you feel have had the most influential in the past"
3) "Which drink or drinks do you think others would see as currently the most
influential"
4) "Which drinks do you think others would see as the most influential in the
past".
I am not sure about the 'do you feel' part in question one and two. They tend
to be more representative for 'popularity' or 'perceived influence'. Therefore
I've added questions three and four. If the results of the answers are too
different you may see the questionnaire as invalid since too many other
factors like popularity might have played a role. The division between 1
present and 2 past is because general influence is difficult to grasp.
Questioning people about their idea of the most influential drinks is not
necessarily coinciding with the most influential drinks. If you define the
question a little bit more specific and less abstract than the quantity the
result is representing may be shifting a little bit more from the quantity
'perceived influence' towards the quantity 'true influence'.
Some examples of more specific questions:
5) Drinks which have influenced me personally the most
6) Drinks which have influenced and the people around me the most
7) Drinks which have had the most influence on the histroy of drinks.
- Control questions -
Example: If you would like to do a research about the influence of alcohol
consumption and the life expectancy you can not simply measure the alcohol
consumption and the age of dead and use the correlation between these two.
People who often drink could, for instance,also be smoking more often. In
order to rule out the effect of the correlation to be due to the smoking one
has to measure the smoking habits as well.
A single question can not be enough to represent a proper quantity. From your
questionnaire it becomes not clear what the average bartender thinks because
you don't know whether the people who have been filling in your questionnaire
are your average bartender.
For your research I would measure age (younger people might be less acquainted
with some drinks), location (not every drinks is equally important from place
to place), bartending experience.
Using those variables you might check if they have influence and you should
compensate if the distribution of your voters among the groups is not
representing the average bartender.
An interesting extra question could be which drinks people know and which not.
While famousness of a drink might indicate influence, some influential
cocktails may have been slightly forgotten. (because of this also the
difference between question 1 and 2). You might want to increase the score of
a drink which is less well known as this 'being less well know' has influence
on it's score but not necessarily because of a lower influence. You could
present this as a separate result or take into account when making conclusions
about the result. Again, you should not want to grasp influence by means of a
single question and a single value.
- question form -
The way people can fill in a question form can have influence on the result.
For instance the number of points on a scale. If this is an even number of
points people can not fill in a middle option and are forced to decide between
left or right.
It is a good thing that you have abandoned the one vote only thing (and also
changed alphabetic order to random order). Drinks which have much resemblance
with other drinks have a disadvantage. People need to make a choice between
those drinks while people who are voting for a sort of drink of which there
are few with a lot resemblance do not have to make such choice.
Still, also if you have people using more votes there is still an effect.
There are for instance many martini type of drinks. When people are ticking
the options they might think something like 'I already have a lot of these
type of drinks' and vote less for the drinks which resemble each other a lot.
(I predict loners like Bramble, Bloody Mary and Kamikaze will score high. A
drink like French 75 will score low because of several other champagne drinks
on the list)
It is best to have a voting system in which the determination of the influence
of a particular drink is not influenced by other drinks on the list. Such a
decoupled system would be when for each drink people had to fill in a scale
from 'very influential' to 'very not influential' (you could also add the
option "don't know").
Such system would also introduce graduality in the answers. Now it is only yes
and no. People are forced to make a strong point. There is no measurement of
the finer graduation. If you would be searching for a top 5 or top 10 this
wouldn't be a big problem. But since you are looking for a top 25, in which
the difference between the drinks in the lower regions are fine, you should be
having a finer scale in order to measure peoples opinion about drinks.
- Statistical tests -
An almost final remark: You can not automatically see your results as a rating
which is a one to one relation with the top 25 ranking. Your questionnaire is
subject to chance. Your results could be coincidental and do not need to
reflect an underlying ranking. I would calculate whether the differences are
by any reasonable chance neglectible. If they are than the difference should
not be regarded as a true difference. I personally always detest those
rankings in which the differences aren't of any meaning but the ranking is
still being made according to those meaningless differences.
Also very important: if you would be using a sort of scoring system like I
described, it is good to check if the votes represent any natural
distribution. For a natural distribution, e.g. the Gaussian distribution, the
average has a clear meaning. However if the distribution is not natural (e.g.
only low and high scores) than it is very difficult to give any clear meaning
to the average and it should actually not be used. It is often seen in
internet polls that you have a lot people giving high votes and a lot people
giving low votes. The average of those votes isn't really an average of some
value but, instead, it represents more something like the rate between people
voting 0 and people voting 10. You should doubt the validity of your
questionnaire in this case.
Final point: Imagine that the influence of a drink can not be grasped by a
single value but should more extensively be explained from different
viewpoints with more different possible values playing a role. If we create a
certain questionnaire whose result is a single number (so the creator defines
how the extensive body of complexity must be reduced into a single number)
than we might have more influence on the final result than the opinions of the
people filling in that questionnaire. It happens very very often that you read
in a magazine or a paper about some questionnaire results while the it is
practically the structure and the design of the questionnaire which influences
the results instead of the people filling in the questionnaire.