Skip navigation
Skip navigation

Improved Generalization Through Explicit Optimization of Margins

Mason, Llew; Bartlett, Peter; Baxter, Jon

Description

Recent theoretical results have shown that the generalization performance of thresholded convex combinations of base classifiers is greatly improved if the underlying convex combination has large margins on the training data (i.e., correct examples are classified well away from the decision boundary). Neural network algorithms and AdaBoost have been shown to implicitly maximize margins, thus providing some theoretical justification for their remarkably good generalization performance. In this...[Show more]

dc.contributor.authorMason, Llew
dc.contributor.authorBartlett, Peter
dc.contributor.authorBaxter, Jon
dc.date.accessioned2015-12-13T23:16:42Z
dc.identifier.issn0885-6125
dc.identifier.urihttp://hdl.handle.net/1885/89538
dc.description.abstractRecent theoretical results have shown that the generalization performance of thresholded convex combinations of base classifiers is greatly improved if the underlying convex combination has large margins on the training data (i.e., correct examples are classified well away from the decision boundary). Neural network algorithms and AdaBoost have been shown to implicitly maximize margins, thus providing some theoretical justification for their remarkably good generalization performance. In this paper we are concerned with maximizing the margin explicitly. In particular, we prove a theorem bounding the generalization performance of convex combinations in terms of general cost functions of the margin, in contrast to previous results, which were stated in terms of the particular cost function sgn(θ-margin). We then present a new algorithm, DOOM, for directly optimizing a piecewise-linear family of cost functions satisfying the conditions of the theorem. Experiments on several of the datasets in the UC Irvine database are presented in which AdaBoost was used to generate a set of base classifiers and then DOOM was used to find the optimal convex combination of those classifiers. In all but one case the convex combination generated by DOOM had lower test error than AdaBoost's combination. In many cases DOOM achieves these lower test errors by sacrificing training error, in the interests of reducing the new cost function. In our experiments the margin plots suggest that the size or the minimum margin is not the critical factor in determining generalization performance.
dc.publisherKluwer Academic Publishers
dc.sourceMachine Learning
dc.subjectKeywords: Costs; Data structures; Database systems; Decision theory; Functions; Learning algorithms; Neural networks; Optimization; Theorem proving; Boosting methods; Margins analysis; Voting methods; Learning systems
dc.titleImproved Generalization Through Explicit Optimization of Margins
dc.typeJournal article
local.description.notesImported from ARIES
local.description.refereedYes
local.identifier.citationvolume38
dc.date.issued2000
local.identifier.absfor080105 - Expert Systems
local.identifier.ariespublicationMigratedxPub19600
local.type.statusPublished Version
local.contributor.affiliationMason, Llew, College of Engineering and Computer Science, ANU
local.contributor.affiliationBartlett, Peter, College of Engineering and Computer Science, ANU
local.contributor.affiliationBaxter, Jon, College of Engineering and Computer Science, ANU
local.description.embargo2037-12-31
local.bibliographicCitation.issue3
local.bibliographicCitation.startpage243
local.bibliographicCitation.lastpage255
local.identifier.doi10.1023/A:1007697429651
dc.date.updated2015-12-12T08:49:17Z
local.identifier.scopusID2-s2.0-0033870982
CollectionsANU Research Publications

Download

File Description SizeFormat Image
01_Mason_Improved_Generalization_2000.pdf116.16 kBAdobe PDF    Request a copy


Items in Open Research are protected by copyright, with all rights reserved, unless otherwise indicated.

Updated:  19 May 2020/ Responsible Officer:  University Librarian/ Page Contact:  Library Systems & Web Coordinator