Pac996

I can't wait to see my car isn't as good as a honda accord

mkaresh

It's only a solution if it happens. I'm not holding my breath on this one.

Cars would be much better if they performed like a 911 turbo, had the gas mileage of a Prius, and the price of a Nano.

mkaresh

I'm pretty sure it does at least some things better than an Accord.

And one of the main things I try to communicate, hard as communication tends to be, is that even when a car is "worse than average" we're still usually talking about less than one repair trip a year. Few people realize how low the average is, because no one else has been providing the actual repair frequencies for each model.

Dots lead people to think that the differences between cars are larger than they actually are.

s_kelly

"See post #27 and you'll see why no matter what the OP says, the endeavor is still statistically flawed. "

Post 27 was the post I was specifically responding to. Not all surveys are the kind they conduct in college psych classes. Not all surveys try to accurately predict the number of xyz valves that will fail. Some just try to get a general direction - up or down. For example, if the survey included something along the lines of average mileage between from rear tire changes, you'd get quite a variance, but you'd probably see a good cluster (likely under 20k miles) wiith a few outliers. While I agree that this not this would pass muster at a in the legal or academic world, it would likely agree with the user-experiences I've seen on rennist & might be helpful to other 996 owners or prospective newbies.

mkaresh

With the current sample sizes, the goal is in between those two extremes. I'm not about to claim that we're measuring repair frequencies within 5-10 repair trips per 100 cars. But we can say whether the frequency is roughly 30, 60, 90, 120, or so forth.

I have a Ph.D. from the University of Chicago that involved a fair amount of training in survey research and statistical analysis. There's a chance I know what I'm doing.

Pac996

If you have a 911 then you might understand only compairing it to other cars that take in town turns at 30-40 with hardly ever a squak from the tires. Kind of adds to the abilty to take wear and tear in all departments. Then the total revs are generally the life span of an engine with the high revs giving birth to faster wear and tear. I kind of wonder how a stock 911 would hold up in a stock car race. It would be kind of funny to see a porsche at daytona. Might be too insulting to detroit engineering to run a bone stock car and clean up. I'm sure the racers commo would normally be traffic like "save me baby jesus!"

BruceP

You're dealing with a group of people that is very sensitive to bad and anecdotal public information about their cars, so you're going to be held to a higher confidence standard than whatever you think is 'directionally valid' (a term I've heard about a million times in 30 years of working with market research, and which usually follows some kind of wild-*** observation based on a fragment of questionable data). If you're experienced with this stuff, then you know that every additional compromise in the sample design multiplies the degree to which confidence is reduced. You have three: A potentially questionable sample, plus a self-selecting sample, plus longitudinal self-reporting. Any one of these is an asterisk, but all three is sketchy.

The last thing anybody here wants is a piece of data that misrepresents the cost of owning a 996, but which floats around out there on the interweb doing a great imitation of fact. You should expect a little heat, and not be too dismissive, IMHO.

mkaresh

Where did I say anything about "directionally valid?"

Have you looked at the actual results, and found them questionable, or are your arguments based on textbook theory? I'm aware of the factors you mention, and have been pleasantly surprised by the quality of the actual data. Results could have ended up all over the map. They have not been.

It does help that I selected a metric to minimize variance, and am not relying on respondents to report much beyond whether or not the car required and repair, and whether this repair was successful.

I'm well aware that many people would prefer no results to a result they do not care for. I have not found a good way to respond to this. I personally prefer imperfect information to no information. In the real world, perfect information is rarely possible.

Ironically, I started trying to get more Porsche owners involved because Porsche owners kept contacting me to.

BruceP

My arguments are not based on 'textbook theory'. I'm not the one who trotted out his PhD. My arguments are based on decades of experience in interpreting market research to advise corporations on making multimillion dollar bets. That's about as 'real world' as it gets.

To your comment about imperfect information, I didn't say that people objected to information they don't like. I said they objected to information that is misleading, which is a risk here.

mkaresh

In your experience advising your clients, are they much more likely to question a study when they don't like the results?

You keep bringing up what might happen. I've been conducting this survey for four years. Can you find any actual cases where people have been misled by my results? Which of the current results do you feel are likely to mislead people?

Here's the full list of the latest stats:

http://truedelta.com/latest_results.php

A few seem off, and are commented as such. This has been much more likely among older cars, I suspect because people start living with problems rather than fixing them. But the great majority of these results are internally consistent and consistent with the results from other surveys. Just with actual repair frequencies, not just dots, and on average nearly a year ahead.

BruceP

No, you're absolutely right. I can't name a single person that has been misled by your results. Therefore, they are immutably true.

(This same logic was used to great success in reporting Polish crop yields to the Soviet government.)

Dude, seriously. That's a pretty lame response. As is the fact that your research is validated by its resemblance to the research of others (raising the question of why it's worth doing in the first place). To answer your first question, my clients challenge research on the basis of its design. I imagine they would be especially skeptical if the data featured banner ads and Google AdSense links. If you're going to use a forum like this to drive traffic to your commercial web site, then I think it's perfectly reasonable to ask you to defend your product... and kind of nice of me to warn you that it's a sore point around here.

Anyway, good luck with it.

mkaresh

I did not say that since you cannot name anyone who has been misled, then no one has been misled. Why twist what I said? I was merely trying to shift the discussion from hypotheticals to what has actually been happening.

I do not see what Adsense (also the source of any banner ads) has to do with anything here. Conducting this research is more than a full time job for me, about 70 hours a week, and I cannot afford to do this for free. The advantage of Adsense is I have absolutely no direct contact with specific advertisers.

I mentioned two key advantages over other studies in my previous post. To repeat:

1. Actual repair frequencies, not just dots--which often ACTUALLY mislead people because few if any people know the repair frequency represented by a dot. So they assume inaccurately.

2. Updated promptly four times a year. Consumer Reports and JD Power currently provide results based on surveys conducted back in the spring of 2009. My current results are based on data through the end of 2009, and will soon cover through the end of March 2010. Would you prefer to know how a car was doing a year ago, when it was a year younger with 12,000 or so fewer miles on it, or how it has been faring recently?

Steve2112

Thanks for your efforts mkaresh - I have been a member on your site with 3 of my past cars, and now with my Porsche. I wish there were more people joining. In fact, after getting a results email from you today, I went to the site to check on Porsche, and saw the note about not having 25 people. I then came here and posted a note to join your site, and then immediately after posting it I saw this thread

Wellardmac

Thanks for your comments Bruce - common sense as usual.

Honestly, how anyone can think that such a small and unrepresentative sample of Porsche owners is worth any more than a steaming pile of BS is beyond me.

No-one is against having good data, but the point that we're making is that mkaresh doesn't have good data at all and is pretending that he does and IS misleading by pretending otherwise. This does no service to anyone.

mkaresh

Thank you for helping me provide a steaming pile of BS

I have no doubt that Bruce is very good at what he does, which is critique the work of others. Anyone who has ever tried to create something knows that the task is much harder than that of critiquing it, and that there is never a shortage of critics.

I am aware of no better publicly available reliability information than the information I am providing. Anyone who thinks they can do better, I invite you to give it a shot.

It is clear to me that the sample sizes are the #1 weakness of my results. As we get more people involved--over 2,000 join each month--this weakness will go away. We already have sample sizes over 100, even 200, for a few cars. I look forward to doing the same for nearly all cars. So I hope anyone interesting in actually doing something, rather than saying what can't be done, will join and help out.