Home People Research Publications Demos
         
News Jobs Prospective
Students
About Internal

Classifier Loss Function Design

The machine learning problem of classifier design is studied from the perspective of probability elicitation, in statistics. This shows that the standard approach of proceeding from the specification of a loss, to the minimization of conditional risk is overly restrictive. It is shown that a better alternative is to start from the specification of a functional form for the minimum conditional risk, and derive the loss function. This has various consequences of practical interest, such as showing that 1) the widely adopted practice of relying on convex loss functions is unnecessary, and 2) many new losses can be derived for classification problems. These points are illustrated by the derivation of a number of novel Bayes consistent loss functions, some of which are not convex but do not compromise the computational tractability of classifier design. A number of algorithms custom tailored for specific classification problems are derived based on novel loss functions. These include classification algorithms for cost sensitive learning, robust outlier resistant classification and variable margin classification.

Algorithms:
SavageBoost
A new boosting algorithm, SavageBoost, is derived for the minimization of the novel Savage loss. This boosting algorithm which is based on the bounded and nonconvex Savage loss function is less sensitive to outliers than conventional methods, such as Ada, Real, or LogitBoost, and converges in fewer iterations.
    

TangentBoost
A new boosting algorithm, TangentBoost, is derived for the minimization of the novel Tangent loss. The bounded and nonconvex Tangent loss function penalizes both large positive and negative margins which results in a much more robust classifier suitable for computer vision datasets with noisy data.
    

Cost Sensitive Boosting
The new cost sensitive AdaBoost, RealBoost and LogitBoost algorithms are derived based on the novel cost sensitive Bayes consistent exponential and logistic losses.
    

Cost Sensitive SVM
The new cost sensitive SVM algorithm is derived based on the novel cost sensitive Bayes consistent hinge loss.
    

Variable Margin Canonical GradientBoost
A detailed analytical study is presented on how properties of the classification risk, such as its optimal link and minimum risk functions, are related to the shape of the loss, and its margin enforcing properties. Novel families of Bayes consistent loss functions, of variable margin, are derived. These families are then used to design boosting style algorithms with explicit control of the classification margin.
    


Publications: Cost-Sensitive Boosting.
Hamed Masnadi-Shirazi and Nuno Vasconcelos
IEEE Trans. on Pattern Analysis and Machine Intelligence,
vol. 32(2), 294, March 2010 .
IEEE [ps] [pdf]

Variable margin losses for classifier design.
Hamed Masnadi-Shirazi and Nuno Vasconcelos.
Neural Information Processing Systems (NIPS), Vancouver, Canada, Dec 2010.
(acceptance rate 25%)
[pdf]

Risk minimization, probability elicitation, and cost-sensitive SVMs
Hamed Masnadi-Shirazi and Nuno Vasconcelos.
International Conference on Machine Learning (ICML), 2010.
(acceptance rate 20%)
[pdf]

On the Design of Robust Classifiers for Computer Vision.
Hamed Masnadi-Shirazi, Nuno Vasconcelos and Vijay Mahadevan.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
Oral presentation (acceptance rate 5%)
CVPR Oral presentation video
[pdf]

On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost.
Hamed Masnadi-Shirazi and Nuno Vasconcelos
Proceedings of Neural Information Processing Systems (NIPS),
Vancouver, Canada, Dec 2008.
[pdf]

Asymmetric Boosting
Hamed Masnadi-Shirazi and Nuno Vasconcelos
Proceedings of International Conference on Machine Learning (ICML),
Corvallis, OR, May 2007.
[pdf]

Contact: Nuno Vasconcelos, Hamed Masnadi-Shirazi

 



Copyright @ 2007 www.svcl.ucsd.edu