Skip navigation
Skip navigation

Visual Permutation Learning

Santa Cruz, Rodrigo; Fernando, Basura; Cherian, Anoop; Gould, Stephen

Description

We present a principled approach to uncover the structure of visual data by solving a deep learning task coined visual permutation learning. The goal of this task is to find the permutation that recovers the structure of data from shuffled versions of it. In the case of natural images, this task boils down to recovering the original image from patches shuffled by an unknown permutation matrix. Permutation matrices are discrete, thereby posing difficulties for gradient-based optimization...[Show more]

dc.contributor.authorSanta Cruz, Rodrigo
dc.contributor.authorFernando, Basura
dc.contributor.authorCherian, Anoop
dc.contributor.authorGould, Stephen
dc.date.accessioned2021-04-14T05:16:51Z
dc.identifier.issn0162-8828
dc.identifier.urihttp://hdl.handle.net/1885/229864
dc.description.abstractWe present a principled approach to uncover the structure of visual data by solving a deep learning task coined visual permutation learning. The goal of this task is to find the permutation that recovers the structure of data from shuffled versions of it. In the case of natural images, this task boils down to recovering the original image from patches shuffled by an unknown permutation matrix. Permutation matrices are discrete, thereby posing difficulties for gradient-based optimization methods. To this end, we resort to a continuous approximation using doubly-stochastic matrices and formulate a novel bi-level optimization problem on such matrices that learns to recover the permutation. Unfortunately, such a scheme leads to expensive gradient computations. We circumvent this issue by further proposing a computationally cheap scheme for generating doubly stochastic matrices based on Sinkhorn iterations. To implement our approach we propose DeepPermNet, an end-to-end CNN model for this task. The utility of DeepPermNet is demonstrated on three challenging computer vision problems, namely, relative attributes learning, supervised learning-to-rank, and self-supervised representation learning. Our results show state-of-the-art performance on the Public Figures and OSR benchmarks for relative attributes learning, chronological and interestingness image ranking for supervised learning-to-rank, and competitive results in the classification and segmentation tasks of the PASCAL VOC dataset for self-supervised representation learning.
dc.description.sponsorshipThis research was supported by the Australian Research Council (ARC) through the Centre of Excellence for Robotic Vision (CE140100016) and was undertaken with the resources from the National Computational Infrastructure (NCI), at the Australian National University (ANU).
dc.format.mimetypeapplication/pdf
dc.language.isoen_AU
dc.publisherIEEE
dc.rights© 2019 IEEE
dc.sourceIEEE transactions on pattern analysis and machine intelligence
dc.subjectpermutation learning
dc.subjectself-supervised learning
dc.subjectrelative attributes
dc.subjectrepresentation learning
dc.subjectlearning-to-rank
dc.titleVisual Permutation Learning
dc.typeJournal article
local.identifier.citationvolume41
dc.date.issued2019
local.publisher.urlhttps://www.ieee.org/
local.type.statusPublished Version
local.contributor.affiliationSanta Cruz, R., ANU College of Engineering & Computer Science, The Australian National University
local.contributor.affiliationFernando, B., ANU College of Engineering & Computer Science, The Australian National University
local.contributor.affiliationGould, S., ANU College of Engineering & Computer Science, The Australian National University
local.description.embargo2099-12-31
dc.relationhttp://purl.org/au-research/grants/arc/CE140100016
local.identifier.essn1939-3539
local.bibliographicCitation.issue12
local.bibliographicCitation.startpage3100
local.bibliographicCitation.lastpage3114
local.identifier.doi10.1109/TPAMI.2018.2873701
CollectionsANU Research Publications

Download

File Description SizeFormat Image
2018 Visual Permutation Learning_Stephen Gould.pdf2.66 MBAdobe PDF    Request a copy


Items in Open Research are protected by copyright, with all rights reserved, unless otherwise indicated.

Updated:  19 May 2020/ Responsible Officer:  University Librarian/ Page Contact:  Library Systems & Web Coordinator