mkaresh

Maintenance and tuning do not count as repairs. A repair is when a problem is present, and then something is done to make that problem go away.

An email goes out each month, so people can respond when their memory is fresh. No repair? Then only respond with an approximate odometer reading at the end of the third month. The form is short, so responding doesn't take much time. If someone doesn't respond for a little while, then resumes, we send a special email to fill the gap.

Last month the overall response rate, including people who joined as much as four years ago, was 37.6%. The response rate for people who joined recently was 55.4%. These are the overall averages--for some models the response rate was much higher. Especially when you factor in that many of the provided email addresses are simply not being monitored, these are very high response rates compared to other such surveys.

My cornflakes are doing just fine.

Bob Rouleau

Ed,

I approved this survey because my intuition is that the alleged problems of the 996 (the M96 engines in general) are exaggerated. Trouble is, in the absence of any meaningful data, everyone assumes the worst. PAG has been no help as we all know. Jake and LN Eng. are certainly helping with cures - a good thing. In the meantime we probably all saw the thread where a guy decided not to buy a nice 996 because he was afraid of an engine failure.

These fears are probably the reason why 996 resale prices are so low. Wouldn't we all like to know the real odds?

I had a 2001 Boxster S. I tracked the heck out of it for 3 years and sold it to a pal who did the same for several more years. He sold it to a pal who is still driving it with no issues. Are 99 percent of the so called M96 engines like that? I'd love to know.

Best,

ivangene

I agree that the fear keeps people from buying the cars... my 99' has ~118k miles on the original everything and is going strong... that said it was like trying to sell a salmon after it had laid its eggs.... not much interest and thus the only way to sell would be to lower the price despite it being an excellent car -

still I have heard that "maybe" around 5% of cars have had issues based on what my indy sees and what we "guess" the number of cars in our area to be is....and for me that is just a number... with out fixing the cars it could be any car and that is the scary part.... untill we start seeing cars with repaired issues going up in value or rumors of cars lasting 200k become the norm... these cars are going to suffer. knowing the data doesnt mean your car wont have it happen (IMO) fixing it before it happens "SEEMS" to be the only way to know for sure -

its a good idea, but I dont see the value in the numbers if there can be no predictions made from it to prevent it from happening, thats all I am saying

Palmbeacher

Are you a health care professional? It doesn't sound like it. The salient issue is an impossibility to verify the data. It doesn't matter if there are ten or a hundred variables, as long as one of them is that you have to take the participant's word without any verification (could be, eg., a lab test or physical examination in the case of a medical study, or shop invoice in the case of a car study). People are people whether in the context of being patients or car owners...and people are sometimes confused or mistaken, and sometimes just plain lie.


That's good. One thing you might consider is requiring a faxed copy of an invoice for major repairs such as exploding engines. Another thing is can you obtain an accurate production number for a particular year. Without that there doesn't seem to be any way to accurately extrapolate percentages even if you can assure the veracity of responses.

tooloud10

The e-mail I got after enrolling says that the 1999 911 is not yet included in the survey...

FWIW, I suspect that each owner's opinion on whether the 996 is "reliable" or not skews heavily one way or the other depending on whether they've ever had to replace the engine or not. I'll bet I've spent more on repairs to keep my Porsche on the road than I have on every other car I've ever owned combined.

Bob Rouleau

Jake, the survey justs asks whether the car has needed a repair. Reliability is estimated based on the need or not for repairs. There is no opinion involved.

Michael will add models as soon as he has enough data I believe. He can best answe this mind you

mkaresh

It is important to keep it simple. I would not attempt to learn anything too complicated by means of these surveys. But whether or not the car was in the shop because of a problem, and whether or not the shop was able to do something to make the problem go away? People can easily handle this.

Note that also, unlike many medical studies, I'm not asking people about something where they have to choose between lying and admitting that they did something they know they shouldn't have been doing. I'm not asking them how many beers they drink on the typical night, what illegal drugs they're taking, whether they've been avoiding fatty foods, or regularly flossing their teeth.

Requiring a fax would probably reduce the number of responses by at least 90%--the cure would be far worse than the disease. Again, with the methods I use I don't think we're missing much. The number of repairs people forget about initially and then have to report later is surprisingly small.

It is critical that the emails go out once a month. With an annual survey of the sort CR and JD Power conduct people will forget to report a fairly high percentage of repairs. I've seen a lot of evidence for this.

There's no need to know the size of the population to calculate percentages.

Shark Attack

But another issue is, a lot of his data is going to be off of the real numbers. He needs the info from the dealers. Not just our little internet community. To show you how little... Have you EVER passed a lister on the street? or even talked to one?

And where do people go for info when they do have a problem???? The internet.. I am afraid to say, if all his info is just from the internet and he doesnt want to invest on hiring a company to travel the country, get mailing lists and everything else, his ratio is going to be WAY HIGH on failures. He is not going to get an accurate count... To do this he would need to invest 100's of 1000's maybe more. I am sorry to say. Its a good effort but its kinda like beating someone to death with a loaded uzi.

mkaresh

Bob already responded to the "opinion" bit.

The number of cars to start, and more is certainly better, is 25. The 1999, 2001, 2002, and 2003 are now all about halfway. Starting before enough cars are signed up would risk not being able to use the data people took the time to provide, so I don't do it.

mkaresh

I've heard this argument countless times. It's a hypothesis that makes a lot of logical sense, but one that is overwhelmingly rejected by the actual data. Among newer cars, the average reported repair frequency is about half a repair trip per car per year, and for the best cars it's about a tenth of a repair trip per car per year. If you look at the actual results, I think you'll agree that they're far from "way high."

I do think it makes a big difference that data is collected continuously going forward, and that people can't respond the instant they sign up. This survey is structured differently than other surveys for a reason.

tooloud10

Thanks for the clarification, guys.

I don't personally understand the uproar myself. If you're not interested, don't sign up. If you don't trust the data, why not just ignore it? I'll participate, but life's too short to worry about the reliability of a fancy car. I've almost forgotten that my 996's engine blew at 40k miles.

Palmbeacher

No matter how little you attempt to learn, the issue still remains whether or not the data is reliable...and if it's un-verified, it's not reliable.

You aren't asking them to specify their driving and maintenance habits??!! Those two are hugely important variables...and ones that people are not unlikely to lie about.

You missed my point. I wasn't concerned with missing repairs, I was concerned that if someone reports his engine blew from IMS failure, that he substantiates it with a repair invoice. Otherwise what's to stop one person from using a bunch of different e-mail addresses to bump up the numbers of, just as an hypothetical example, IMS bearing failure?

Of course it's possible mathematically. However it's invalid, if not downright reckless for you to extrapolate and generalize beyond your cohort. Your data is completely un-verified, and your cohort is neither random nor adjusted for variables. You don't even have positive proof your respondents even own(ed) the cars they're reporting on, let alone that what they report is factual. True you don't have the life/death liability you would if testing a drug, but the results of your survey could have a profound effect on the resale value of these autos, so I think there's a compelling ethical argument to be made against playing fast and loose with methodology.

LVDell

Actually you have confused reliability (and I have no doubt his measure is reliable) with validity. In this case, it's validity, more specifically construct validity and criterion (predictive) validty, that is of concern.

mkaresh

Analysis of survey data ALWAYS extrapolates from the sample to the population. It's the entire point. I'm not sure how you think the actual number of cars in the population might play into this--it's irrelevant.

I run a large number of checks for errors in the data, duplicate cars, and so forth.

After doing this for four years and about 40,000 reported repairs, I have yet to come across even a shred of evidence that anyone has made anything up in an attempt to influence the results.

I always find a small number of duplicate entries--one to two tenths of a percent of the total--but these are generally because people don't realize that they've already enrolled their car once and then enroll it again.

Under-reporting is far more of a concern than over-reporting. And it happens predominately because the average memory is poor and people "never get around to it." The challenge is to get people to respond for the cars they do have, not preventing them from responding for cars they don't have. The survey doesn't take much time, but it's not a ten-seconds-and-done-forever survey like many on the Internet. This might screen out some of the behavior you're concerned about.

Lastly, I find it highly unlikely that this survey will have a significant impact on resale values in either direction. It's far from the only source of reliability information.

So forgive me if I don't spend millions of dollars I don't have to fix what seems to be a purely hypothetical problem

BruceP

Wait, I thought that was the point. I thought this was a heroic effort to set the record straight about the reliability of 996s.