exit poll redux
18 November 2004 10:14A friend's blog asks which of the following two studies of exit polls should be regarded as reliable:
"this guy says the exit polls couldn't have been off so far by chance" (Stephen Freeman, UPenn)
"these guys say the difference is insignificant" (CalTech/MIT Voting Technology Project)
So, I'm not much of a statistician; I understand some things about probability and I speak a few distributions and so on, but I'm not accustomed to doing confidence analyses. So I'd have to do a lot more scut work to check on the analyses themselves.
Instead, I can provide a higher-level meta-analysis, which might be of interest.
The CalTech/MIT analysis doesn't really give me much confidence in their results: they don't go through their methods, they don't talk about exactly what data they're working with or where they got it from, and they don't really address possible confounding factors very well. The whole analysis strikes me as being kind of shallow, and there isn't enough here for me to be able to check their work. I also have the vague feeling that they basically bought the "exit polls were skewed" argument coming out the door.
By contrast, Freeman's analysis (after a brief skim) seems to have none of these problems. I can't guarantee that he didn't screw something up, but at least I feel that I could back-check him if I wanted to take the time.
Update: Robin has given me permission to post our e-mail discussion about this. I'll be posting my future responses, if any, in the comments.
( Robin's first response to the above, on the nature of the data )
( My recognition of a difference in aims )
( Robin's response to my identification of a dichotomy )
"this guy says the exit polls couldn't have been off so far by chance" (Stephen Freeman, UPenn)
"these guys say the difference is insignificant" (CalTech/MIT Voting Technology Project)
So, I'm not much of a statistician; I understand some things about probability and I speak a few distributions and so on, but I'm not accustomed to doing confidence analyses. So I'd have to do a lot more scut work to check on the analyses themselves.
Instead, I can provide a higher-level meta-analysis, which might be of interest.
The CalTech/MIT analysis doesn't really give me much confidence in their results: they don't go through their methods, they don't talk about exactly what data they're working with or where they got it from, and they don't really address possible confounding factors very well. The whole analysis strikes me as being kind of shallow, and there isn't enough here for me to be able to check their work. I also have the vague feeling that they basically bought the "exit polls were skewed" argument coming out the door.
By contrast, Freeman's analysis (after a brief skim) seems to have none of these problems. I can't guarantee that he didn't screw something up, but at least I feel that I could back-check him if I wanted to take the time.
Update: Robin has given me permission to post our e-mail discussion about this. I'll be posting my future responses, if any, in the comments.
( Robin's first response to the above, on the nature of the data )
( My recognition of a difference in aims )
( Robin's response to my identification of a dichotomy )