DM: FYI: prediction accuracy of classification trees

[Tjen-Sien Lim]

I've completed the prediction accuracy part of my review. The datasets used in the experiment can be downloaded from http://www.stat.wisc.edu/~limt/mv.html

Note that since there's a design flaw in the prediction part of SPSS AnswerTree, the results for AnswerTree QUEST don't reflect the true predictive performance of the QUEST algorithm.

Error Rates

--------------------------------------------------------------------------------------------
         Plurality          ARCed  Bagged  KnowledgeSEEKER See5    Boosted  See5 Boosted  AnswerTree
Datasets   Rule    CARTŪ (r) CARTŪ (r) CARTŪ (r)  (EXHAUSTIVE)   Tree   See5 Tree Rule See5 Rule (QUEST)
-----------------------------------------------------------------------------------------------------

adt        .236     .142    .155    .151        .174       .149     .150    .148   .152      .172
att        .496     .394    .397    .396        .377       .398     .396    .394   .383      .461
ban        .422     .322    .195    .217        .329       .259     .200    .254   .172      .290
bcw        .345     .0586   .0344   .0459       .0466      .0644    .0372   .0602  .0373    .0759
bio        .359     .164    .139    .139        .174       .144     .134    .144   .135      .176
bld        .419     .334    .293    .288        .390       .316     .278    .309   .269      .420
bos        .657     .256    .218    .212        .292       .229     .202    .223   .210      .267
bpr        .397     .313    .251    .218        .409       .339     .254    .342   .237      .414
cmc        .573     .447    .500    .479        .466       .488     .492    .481   .484      .476
crx        .445     .149    .135    .138        .166       .146     .139    .146   .129      .162
der        .694     .0460   .0354   .0319       .0354      .0680    .0192   .0674  .0217     .255
ech        .328     .358    .349    .351        .328       .378     .357    .378   .349      .348
edu        .461     .436    .429    .423        .445       .454     .456    .424   .445      .477
hab        .265     .261    .334    .308        .278       .288     .268    .288   .265      .304
hco        .369     .169    .185    .163        .137       .160     .163    .163   .161      .313
hea        .459     .221    .204    .211        .214       .281     .195    .256   .191      .254
hep        .206     .233    .174    .175        .246       .188     .155    .176   .161      .742
hin        .491     .279    .300    .290        .302       .293     .258    .281   .256      .437
hyp        .0477    .00727  .0126   .00980      .0136      .00759   .00885  .00759 .00917   .0177
imp        .672     .233    .142    .179        .369       .225     .139    .237   .164      .410
pid        .349     .245    .249    .238        .284       .256     .248    .249   .241      .262
usn        .660     .279    .232    .236        .341       .286     .235    .283   .243      .301
tae        .656     .365    .352    .352        .497       .503     .477    .503   .551      .558

Means:
Error Rate          .248    .231    .228        .275       .257     .229    .253   .229      .330
Rank                4.85    4.37    3.76        6.26       6.13     3.37    5.35   2.96      7.96

Multiple comparisons: 
1. Two methods are statistically significantly different at 10% simultaneous level when their means error rate differ by at least 0.0424.
2. Two methods are statistically significantly different at 10% simultaneous level when their means rank differ by at least 2.33.

-- 
Tjen-Sien Lim                (608) 262-8181                        
Dept. of Statistics          limt@stat.wisc.edu
Univ. of Wisconsin-Madison   http://www.stat.wisc.edu/~limt
1210 West Dayton Street       
Madison, WI 53706