From chemistry-request@ccl.net Thu Apr 14 14:55:30 2005 Received: from mx5.informatik.uni-tuebingen.de (mx5.Informatik.Uni-Tuebingen.De [134.2.12.32]) by server.ccl.net (8.13.1/8.13.1) with ESMTP id j3EItOVK027924 for ; Thu, 14 Apr 2005 14:55:24 -0400 Received: from localhost (loopback [127.0.0.1]) by mx5.informatik.uni-tuebingen.de (Postfix) with ESMTP id B4D2610E; Thu, 14 Apr 2005 18:57:57 +0200 (DFT) Received: from mx3.informatik.uni-tuebingen.de ([134.2.12.26]) by localhost (mx5 [134.2.12.32]) (amavisd-new, port 10024) with ESMTP id 42468-02; Thu, 14 Apr 2005 18:57:55 +0200 (DFT) Received: from [134.2.10.184] (rapc84.informatik.uni-tuebingen.de [134.2.10.184]) by mx3.informatik.uni-tuebingen.de (Postfix) with ESMTP id 3C870149; Thu, 14 Apr 2005 18:57:53 +0200 (DFT) Message-ID: <425EA110.1000207^at^informatik.uni-tuebingen.de> Date: Thu, 14 Apr 2005 18:57:52 +0200 From: "Joerg K. Wegner" Organization: Department of Computer Architecture User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.3) Gecko/20040910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: lhhu^at^yangtze.hku.hk Cc: chemistry^at^ccl.net Subject: Re: CCL:Correlation Coefficient References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: by amavisd-new (McAfee AntiVirus) at informatik.uni-tuebingen.de X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=failed version=3.0.1 X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on server.ccl.net Hi Holly, I would say there is no general answer and values of 0.8 or whatever seems suspect to me. Why? The quality depends highly on the data set used, even on data set with drug molecules [1]. Furthermore, if you apply an optimization algorithm to select a better set of features/descriptors, better induction parameters, etc. you can easily produce an overfitted result [2] (see references therein). So, there is no short answer, but yes you can use the cross-validation result (of the outer loop) to estimate the future performance of your hypothesis. In other words, if you have an infinite amount of data the true error is the same as the experimental error. In all other cases with a finite number of training molecules it is not. I would recommend to compare your result to other method based approaches, which is the most serious way to compare prediction rates. But as already said in [2] you should not use leave-one-out cross-validation for comparing feature selection methods, also not emprical selected feature sets. [1] Wolpert, D. H. and Macready, W. G. No Free Lunch Theorems for Optimization IEEE Transactions on Evolutionary Computation, 1997 , 1 , 67-82 [2] Wegner, J. K. and Frvhlich, H. and Zell, A. Feature selection for Descriptor based Classification Models. 2. Human Intestinal Absorption (HIA) J. Chem. Inf. Comput. Sci., 2004 , 44 , 931-939 Kind regards, Joerg > I have a question about Correlation Coefficient of cross-validation. Can > anyone tell me how much the value of Correlation Coefficient indicate > the result is reliable? Certainly I know the larger the value, the more > reliable the result. I want to know the lowest limit of that value. I > read some papers that used the Correlation Coefficient value, some > values are lower than 0.8, is it OK for present them in journals? Or it > depends on different situation or topic? > > Thanks a lot to you all, > > Holly > > > > > > -= This is automatically added to each message by the mailing script =- > To send e-mail to subscribers of CCL put the string CCL: on your > Subject: line > and send your message to: CHEMISTRY^at^ccl.net > > Send your subscription/unsubscription requests to: > CHEMISTRY-REQUEST^at^ccl.net HOME Page: http://www.ccl.net | Jobs Page: > http://www.ccl.net/jobs > If your is mail bouncing from ccl.net domain due to spam filters, please > use the Web based form from CCL Home Page > -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > > > > > -- Dipl. Chem. Joerg K. Wegner Center of Bioinformatics Tuebingen (ZBIT) Department of Computer Architecture Univ. Tuebingen, Sand 1, D-72076 Tuebingen, Germany Phone: (+49/0) 7071 29 78970 Fax: (+49/0) 7071 29 5091 E-Mail: mailto:wegnerj^at^informatik.uni-tuebingen.de WWW: http://www-ra.informatik.uni-tuebingen.de -- Never mistake motion for action. (E. Hemingway) Never mistake action for meaningful action. (Hugo Kubinyi,2004)