Topics in Convex Optimization: Interior-Point Methods,

Conic Duality and Approximations

François Glineur

Service de Mathématique et de Recherche Opérationnelle,

Faculté Polytechnique de Mons, Rue de Houdain, 9, B-7000 Mons, Belgium.

             Francois.Glineur@fpms.ac.be
          http://mathro.fpms.ac.be/~glineur/

January 2001 Co-directed by

Jacques Teghem Tamás Terlaky

Contents

Table of Contents i

List of figures v

Preface vii

Introduction 1

I Interior-Point Methods 5

1 Interior-point methods for linear optimization 7

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1.1 Linear optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1.2 The simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1.3 A first glimpse on interior-point methods . . . . . . . . . . . . . . . . 9 1.1.4 A short historical account . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2 Building blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2.1 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2.2 Optimality conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.2.3 Newton’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.2.4 Barrier function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2.5 The central path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2.6 Link between central path and KKT equations . . . . . . . . . . . . . 15

1.3 Interior-point algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.1 Path-following algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.2 Affine-scaling algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.3.3 Potential reduction algorithms . . . . . . . . . . . . . . . . . . . . . . 25

1.4 Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1.4.1 Infeasible algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1.4.2 Homogeneous self-dual embedding . . . . . . . . . . . . . . . . . . . . 27 1.4.3 Theory versus implemented algorithms . . . . . . . . . . . . . . . . . . 29 1.4.4 The Mehrotra predictor-corrector algorithm . . . . . . . . . . . . . . . 29

1.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.5.1 Linear algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.5.2 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 1.5.3 Starting point and stopping criteria . . . . . . . . . . . . . . . . . . . 33

1.6 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2 Self-concordant functions 35

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.1.1 Convex optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.1.2 Interior-point methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.1.3 Organization of the chapter . . . . . . . . . . . . . . . . . . . . . . . . 38

2.2 Self-concordancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.2.2 Short-step method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.2.3 Optimal complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.3 Proving self-concordancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.3.1 Barrier calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.3.2 Fixing a parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.3.3 Two useful lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

2.4 Application to structured convex problems . . . . . . . . . . . . . . . . . . . . 54 2.4.1 Extended entropy optimization . . . . . . . . . . . . . . . . . . . . . . 54 2.4.2 Dual geometric optimization . . . . . . . . . . . . . . . . . . . . . . . 55 2.4.3 l_{p}-norm optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.5 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

II Conic Duality 59

3 Conic optimization 61 3.1 Conic problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 3.2 Duality theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.3 Classification of conic optimization problems . . . . . . . . . . . . . . . . . . 67

3.3.1 Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.3.2 Attainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.3.3 Optimal duality gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4 l_{p}-norm optimization 73

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.1.1 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.1.2 Organization of the chapter . . . . . . . . . . . . . . . . . . . . . . . . 75

4.2 Cones for l_{p}-norm optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.2.1 The primal cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.2.2 The dual cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.3 Duality for l_{p}-norm optimization . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.3.1 Conic formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.3.2 Duality properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.3.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

4.4 Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.5 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5 Geometric optimization 95 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.2 Cones for geometric optimization . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.2.1 The geometric cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.2.2 The dual geometric cone . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.3 Duality for geometric optimization . . . . . . . . . . . . . . . . . . . . . . . . 103 5.3.1 Conic formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.3.2 Duality theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.3.3 Refined duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3.4 Summary and examples . . . . . . . . . . . . . . . . . . . . . . . . . . 113

5.4 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.4.1 Original formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.4.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

6 A different cone for geometric optimization 119 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.2 The extended geometric cone . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6.3 The dual extended geometric cone . . . . . . . . . . . . . . . . . . . . . . . . 122 6.4 A conic formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

6.4.1 Modelling geometric optimization . . . . . . . . . . . . . . . . . . . . . 125 6.4.2 Deriving the dual problem . . . . . . . . . . . . . . . . . . . . . . . . . 126

6.5 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

7 A general framework for separable convex optimization 129 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 7.2 The separable cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 7.3 The dual separable cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 7.4 An explicit definition of K^{f} . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 7.5 Back to geometric and l_{p}-norm optimization . . . . . . . . . . . . . . . . . . . 136 7.6 Separable convex optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 138 7.7 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

III Approximations 143

8 Approximating geometric optimization with l_{p}-norm optimization 145 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 8.2 Approximating geometric optimization . . . . . . . . . . . . . . . . . . . . . . 146

8.2.1 An approximation of the exponential function . . . . . . . . . . . . . . 146 8.2.2 An approximation using l_{p}-norm optimization . . . . . . . . . . . . . . 147

8.3 Deriving duality properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 8.3.1 Duality for l_{p}-norm optimization . . . . . . . . . . . . . . . . . . . . . 149 8.3.2 A dual for the approximate problem . . . . . . . . . . . . . . . . . . . 150 8.3.3 Duality for geometric optimization . . . . . . . . . . . . . . . . . . . . 152

8.4 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

9 Linear approximation of second-order cone optimization 155 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 9.2 Approximating second-order cone optimization . . . . . . . . . . . . . . . . . 157

9.2.1 Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 9.2.2 Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 9.2.3 A first approximation of 𝕃^{2} . . . . . . . . . . . . . . . . . . . . . . . . 159 9.2.4 A better approximation of 𝕃^{2} . . . . . . . . . . . . . . . . . . . . . . . 160 9.2.5 Reducing the approximation . . . . . . . . . . . . . . . . . . . . . . . 164 9.2.6 An approximation of 𝕃^{n} . . . . . . . . . . . . . . . . . . . . . . . . . . 166 9.2.7 Optimizing the approximation . . . . . . . . . . . . . . . . . . . . . . 167 9.2.8 An approximation of second-order cones optimization . . . . . . . . . 170 9.2.9 Accuracy of the approximation . . . . . . . . . . . . . . . . . . . . . . 171

9.3 Computational experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 9.3.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 9.3.2 Truss-topology design . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 9.3.3 Quadratic optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 181

9.4 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

IV Conclusions 187

Concluding remarks and future research directions 189

V Appendices 191

A An application to classification 193 A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 A.2 Pattern separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 A.3 Maximizing the separation ratio . . . . . . . . . . . . . . . . . . . . . . . . . . 196 A.4 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

B Source code 201

Bibliography 209

Summary 215

About the cover 217

List of Figures

2.1 Graphs of functions r_{1} and r_{2} . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

3.1 Epigraph of the positive branch of the hyperbola x_{1}x_{2} = 1 . . . . . . . . . . . 68

4.1 The boundary surfaces of L^{(5)} and L^{(2)} (in the case n = 1). . . . . . . . . . . 77 4.2 The boundary surfaces of L^{( \frac{5}{4} )} and L^{(5)} (in the case n = 1). . . . . . . . . . . 81

5.1 The boundary surfaces of G^{2} and (G^{2})^{∗}. . . . . . . . . . . . . . . . . . . . . . 102

9.1 Approximating B_{2}(1) with a regular octagon. . . . . . . . . . . . . . . . . . . 160 9.2 The sets of points P_{3}, P_{2}, P_{1} and P_{0} when k = 3. . . . . . . . . . . . . . . . . 162 9.3 Constraint matrices for L_{15} and its reduced variant. . . . . . . . . . . . . . . 165 9.4 Linear approximation of a parabola using L_{k} for k = 1, 2, 3, 4. . . . . . . . . . 172 9.5 Size of the optimal approximation versus accuracy (left) and dimension (right). 176

A.1 A bidimensional separation problem. . . . . . . . . . . . . . . . . . . . . . . . 195 A.2 A separating ellipsoid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 A.3 A simple separation problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 A.4 A pair of ellipsoids with ρ equal to \frac{3}{2}. . . . . . . . . . . . . . . . . . . . . . . 197 A.5 The optimal pair of separating ellipsoids. . . . . . . . . . . . . . . . . . . . . 198 A.6 The final separating ellipsoid. . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Preface

This work is dedicated to my wife, my parents and my grandfather, for the love and support they gave me throughout the writing of this thesis.

First of all, I wish to thank my advisor Jacques Teghem, which understood early that the field of optimization would provide a stimulating and challenging area for my research. Both his guidance and support were crucial in the accomplishment of this doctoral degree. He also provided me with very valuable feedback during the final redaction of this thesis.

A great deal of the ideas presented in this thesis were originally developed during a research stay at the Delft University of Technology which took place in the first half of 1999. I am very grateful to Professors Kees Roos and Tamás Terlaky for their kind hospitality. They welcomed me in their Operations Research department, which provided me with a very stimulating research environment to work in.

Professor Tamás Terlaky accepted to co-direct this thesis. I wish to express him my deep gratitude for the numerous and fruitful discussions we had about my research. Many other researchers contributed directly or indirectly to my current understanding of optimization, sharing with me at various occasions their knowledge and insight about this field. Let me mention Professors Martine Labbé, Michel Goemans, Van Hien Nguyen, Jean-Jacques Strodiot and Philippe Toint, who made me discover some of the most interesting topics in optimization during my first year as doctoral student as well as Professor Yurii Nesterov, who was advisor in my thesis committee.

I also wish to express special thanks to the entire staff of the Mathematics and Operations Research department at the Faculté Polytechnique de Mons, for their constant kindness, availability and support.

I conducted this research as a research fellow supported by a grant from the F.N.R.S. (Belgian National Fund for Scientific Research), which also funded a trip to attend the International Mathematical Programming Symposium 2000 in Atlanta. My research stay at the Delft University of Technology was made possible with the aid of a travel grant awarded by the Communauté Française de Belgique, which also supported a trip to the INFORMS Spring 2000 conference in Salt Lake City.

Mons, December 2000.

Introduction

The main goal of operations research is to model real-life situations where some decisions have to be taken and help to identify the best one(s). One may for example want to choose between several available alternatives, tune numerical parameters in an engineering design or schedule the use of machines in a factory.

The concept of best decision depends of course on the problem considered and is not easy to define mathematically. The most common way to do this is to describe a decision as a set of parameters called decision variables, and try to minimize (or maximize) an objective function depending on these variables. This function may for example compute the cost associated to the decision. Moreover, we are most of the time in a situation where some combinations of parameters are not allowed (e.g. physical dimensions cannot be negative, a system must satisfy some performance requirements, …), which leads us to consider a set of constraints acting on the decision variables.

Optimization is the field of mathematics whose goal is to minimize or maximize an objective function depending on several decision variables under a set of constraints. The main topic of thesis is a special category of optimization problems called convex optimization^{1}.

Why convex optimization ?

A fundamental difficulty in optimization is that it is not possible to solve all problems efficiently. Indeed, it is shown in [Nes96] that a hypothetical method that would be able to handle all optimization problems would require at least 10^{20} operations to solve with 1% accuracy some problems involving only 10 variables. There are basically two fundamentally different ways to react to this distressing fact:

1This class of problems is sometimes called convex programming in the literature. However, following other authors [RTV97, Ren00], we prefer to use the more natural word “optimization” since the term “programming”

is nowadays strongly connected to computer science. The same treatment will be applied to the other classes of problems that will be considered in this thesis, such as linear optimization, geometric optimization, etc.

a. Ignore it, i.e. design a method that can potentially solve all problems. Because of the above-mentioned result, it will be slow (or fail) on some problems, but hopefully will be efficient on most real-world problems we are interested in. This is the approach that generally prevails in the field of nonlinear optimization.

b. Restrict the set of problems that the method is supposed to solve. The goal is then to design a provably efficient method that is able to solve this restricted class of problems. This is for example the approach taken in linear optimization, where one requires the objective function and the constraints to be linear.

Each of these two approaches has its advantages and drawbacks. The major advantage of the first approach is its potentially very wide applicability, but this is counterbalanced by a less efficient analysis of the behaviour of the corresponding algorithms. In more technical terms, methods in first approach can usually only be proven to converge to an optimum (in some weak sense), while one can usually estimate the efficiency of methods designed for special categories of problems, i.e. bound the number of arithmetic operations they need to attain an optimum with a given accuracy. This is what led us to focus our research for this thesis on that second approach.

The next relevant question that has to be answered consists in asking ourselves which classes of problems we are going to study. It is rather clear that there is a tradeoff between generality and algorithmic efficiency: the more general your problem, the less efficient your methods. Linear optimization is in this respect an extreme case: it is a very particular (yet useful) type of problem for which very efficient algorithms are available (see Chapter 1).

However, some problems simply cannot be formulated within the framework of linear programs, which led us to consider a much broader class of problems called convex optimization. Basically, a problem belongs to this category if its objective function is convex and its constraints define a feasible convex set. As we will see in Chapter 2, very effective methods are available to solve these problems.

Unfortunately, checking that a given optimization problem is convex is far from straightforward (and it might even be more difficult than solving the problem itself). We have therefore to consider problems that are designed in a way that guarantees them to be convex. This is done by using specific classes of objective functions and constraints, and is called structured convex optimization. This is the central topic of this thesis, which is treated in Chapters 3–8.

To conclude, we mention that although it is not possible to model all problems of interest with a convex formulation, one can do it in a surprisingly high number of situations, either directly or using a equivalent reformulation. The reward for the added work of formulating the problem as a structured convex optimization problem is the great efficiency of the methods that can be then applied to it.

Overview of the thesis

We give here a short introduction to the research work presented in this thesis, which consists in three parts (we however refer the reader to the abstract and the introductory section placed at the beginning of each chapter for more detailed comments).

a. Interior-point methods. This first part deals with algorithms. We start with the case of linear optimization, for which an efficient method is known since the end of the fifties: the simplex method [Dan63]. However, another class of algorithms that could rival the simplex method was introduced in 1984 [Kar84]: the so-called interior-point methods, which are surveyed in Chapter 1 (this Chapter was published in [Gli98a], which is a translated and reworked version of [Gli97]). These methods can be generalized to handle any type of convex problems, provided a suitable barrier function is known. This is the topic of Chapter 2 [Gli00d], which gives a self-contained overview of the theory of self-concordant barriers for structured convex optimization [NN94].

b. Conic duality. The second part of this thesis is devoted to the study of duality issues for several classes of convex optimization problems. We first present in Chapter 3 conic optimization, a framework to describe convex optimization problems based on the use of convex cones. Convex problems expressed in this fashion feature a very symmetric duality theory, which is also presented in this Chapter. This setting is used in Chapters 4 [GT00] and 5 [Gli99], where we describe and study two classes of structured convex optimization problems known as l_{p}-norm optimization and geometric optimization.

The approach used in these two chapters is very similar: we first define a suitable convex cone that allows us to express our problem with a conic formulation. The properties of this cone are then studied, which allows us to formulate the dual problem. One can then apply the conic duality theory described in Chapter 3 to give simplified proofs of all the duality properties that relate these primal and dual problems. Chapter 4 also presents a polynomial-time algorithm for l_{p}-norm optimization using a suitable self-concordant barrier and the results of Chapter 2.

Despite some similarities, the convex cones introduced in Chapters 4 and 5 do not share the same structure. The goal of Chapter 6 [Gli00b] is to provide a different convex cone for geometric optimization that is more amenable to a common generalization with the cone for l_{p}-norm optimization presented in Chapter 4. This generalization is the topic of Chapter 7, which presents a very large class of so-called separable convex cones that unifies our formulations for geometric and l_{p}-norm optimization, as well as allowing the modelling of several others classes of convex problems.

c. Approximations. The last part of this thesis deals with various approximations of convex problems. Chapter 8 [Gli00a] uncovers an additional connection between geometric and l_{p}-norm optimization by showing that the former can be approximated by the latter. Basically, we are able to associate to a geometric optimization problem a family of l_{p}-norm optimization problems whose optimum solutions tend to the optimal solution of the original geometric problem. This also allows us to derive the duality properties of geometric optimization in a different way. Finally, Chapter 9 [Gli00c] presents computational experiments conducted with the polyhedral approximation of the second-order cone presented in [BTN98]. This leads to a linearizing scheme that allows any second-order cone problem to be solved up to an arbitrary accuracy using linear optimization.

Part I

Interior-Point Methods

CHAPTER 1

Interior-point methods for linear optimization: a guided tour

The purpose of mathematical optimization is to minimize (or maximize) a function of several variables under a set of constraints. This is a very important problem arising in many real-world situations (e.g. cost or duration minimization).

When the function to optimize and its associated set of constraints are linear, we talk about linear optimization. The simplex algorithm, first developed by Dantzig in 1947, is a very efficient method to solve this class of problems [Dan63 ]. It has been thoroughly studied and improved since its first appearance, and is now widely used in commercial software to solve a great variety of problems (production planning, transportation, scheduling, etc.).

However, Karmarkar introduced in 1984 a new class of methods: the so-called interior-point methods [Kar84 ]. Most of the ideas underlying these new methods originate from the nonlinear optimization domain. These methods are both theoretically and practically efficient, can be used to solve large-scale problems and can be generalized to other types of convex optimization problems.

The purpose of this chapter is to give an overview of this rather new domain, providing a clear and understandable description of these methods, both from a theoretical and a practical point of view. This will provide a basis for the following chapters, which will present our contributions to the field.

1.1 Introduction

In this section, we present the standard formulations of a linear program and give a brief overview of the main differences between the simplex method, the traditional approach to solve these problems, and the recently developed class of interior-point methods, as well as a short historical account.

1.1.1 Linear optimization

The purpose of linear optimization is to optimize a linear objective function f depending on n decision variables under a set of linear (equality or inequality) constraints, which can be mathematically stated as (using matrix notation)

min_{x∈ℝ^{n}} f(x) = c^{T} x s.t. \{ \begin{matrix}A_{e}x = b_{e} \\ A_{i}x ≥ b_{i} \end{matrix} ,  (1.1)

where vector x contains the n decision variables, vector c defines the objective function and pairs (A_{e}, b_{e}) and (A_{i}, b_{i}) define the m_{e} equality and m_{i} inequality constraints. Column vectors x and c have size n, column vectors b_{e} and b_{i} have size m_{e} and m_{i} and matrices A_{e} and A_{i} have dimensions m_{e} × n and m_{i} × n.

Many linear programs have simpler inequality constraints, e.g. nonnegativity constraints (x ≥ 0) or bound constraints (l ≤ x ≤ u). The linear optimization standard form is a special case of linear program used for most theoretical developments of interior-point methods:

min_{x∈ℝ^{n}} c^{T} x s.t. \{ \begin{matrix}Ax = b \\ x ≥ 0 \end{matrix} .  (1.2)

The only inequality constraints in this format are nonnegativity constraints for all variables, i.e. there are no free variables (we have thus that m_{i} is equal to n, A_{i} is the identity matrix and b_{i} is the null vector). It is furthermore possible to show that every linear program in the general form (1.1) admits an equivalent program in the standard form, obtainable by adding/removing variables/constraints (by equivalent problem, we mean that solving the transformed problem allows us to find the solution of the original one).

1.1.2 The simplex method

The set of all x satisfying the constraints in (1.2) is a polyhedron in ℝ^{n}. Since the objective is linear, parallel hyperplanes orthogonal to c are constant-cost sets and the optimal solution must be at one of the vertices of the polyhedron (it is also possible that a whole face of the polyhedron is optimal or that no solution exists, either because the constraints defining the polyhedron are inconsistent or because it is unbounded in the direction of the objective function).

The main idea behind the simplex method is to explore these vertices in an iterative way, moving from the current vertex to an adjacent one that improves the objective function value.

This is done using an algebraic characterization of a vertex called a basis. When such a move becomes impossible to make, the algorithm stops. Dantzig proved that this always happens after a finite number of moves, and that the resulting vertex is optimal [Dan63].

1.1.3 A first glimpse on interior-point methods

We are now able to give a first description of interior-point methods. As opposed to the simplex method which uses vertices, these methods start with a point that lies inside the set of feasible solutions. Using the standard form notation (1.2), we define the feasible set P to be the set of vectors x satisfying the constraints, i.e.

P = {x ∈ ℝ^{n} | Ax = b and x ≥ 0} ,

and the associated set P^{+} to be the subset of P satisfying strict nonnegativity constraints

P^{+} = {x ∈ ℝ^{n} | Ax = b and x > 0} .

P^{+} is called the strictly feasible set^{1} and its elements are called strictly feasible points.

Interior-point methods are iterative methods that compute a sequence of iterates belonging to P^{+} and converging to an optimal solution. This is completely different from the simplex method, where an exact optimal solution is obtained after a finite number of steps. Interior-point iterates tend to an optimal solution but never attain it (since the optimal solutions do not belong to P^{+} but to P \ P^{+}). This apparent drawback is not really serious since

⋄ Most of the time, an approximate solution (with e.g. 10^{−8} relative accuracy) is sufficient for most purposes.

⋄ A rounding procedure can convert a nearly optimal interior point into an exact optimal vertex solution (see e.g. [RTV97]).

Another significant difference occurs when an entire face of P is optimal: interior-point methods converge to the interior of that face while the simplex method ends on one of its vertices.

The last difference we would like to point out at this stage is about algorithmic complexity. While the simplex method may potentially make a number of moves that grows exponentially with the problem size [KM72], interior-point methods need a number of iterations that is polynomially bounded by the problem size to attain a given accuracy. This property is with no doubt mainly responsible for the huge amount of research that has been carried out on the topic of interior-point methods for linear optimization.

1.1.4 A short historical account

The purpose of this paragraph is not to be exhaustive but rather to give some important milestones in the development of interior-point methods.

1P^{+} is in fact the relative interior of P, see [Roc70a].

First steps of linear optimization.

1930–1940. First appearance of linear optimization formulations.

1939–1945. Second World War: operations research makes its debuts with mili-

tary applications. 1947. Georges B. Dantzig publishes the first article about the simplex

method for linear optimization [Dan63]. 1970. V. Klee and G. Minty prove that the simplex method has exponential

worst-case complexity [KM72].

First steps of interior-point methods.

1955. K. R. Frisch proposes a barrier method to solve nonlinear programs [Fri55].

1967. P. Huard introduces the method of centers to solve problems with nonlinear

constraints [Hua67]. 1968. A. V. Fiacco and G. P. McCormick develop barrier methods for convex

nonlinear optimization [FM68]. 1978. L. G. Khachiyan applies the ellipsoid method (developed by N. Shor in 1970

[Sho70]) to linear optimization and proves that it is polynomial [Kha79].

It is important to note that these barrier methods were developed as methods for nonlinear optimization. Although they are applicable to linear optimization, their authors do not consider them as viable competitors to the simplex method. We also point out that the complexity advantage of the ellipsoid method over the simplex algorithm is only of theoretical value, since the ellipsoid method turns out to be very slow in practice^{2}.

The interior-point revolution.

1984. N. Karmarkar discovers a polynomial interior-point method that is practically more efficient than the ellipsoid method. He also claims superior performance compared to the simplex method [Kar84].

1994. Y. Nesterov and A. Nemirovski publish a monograph on polynomial

interior-point methods for convex optimization [NN94]. 2000. Since Karmarkar’s first breakthrough, more than 3000 articles have been

published on the topic of interior point methods. A few textbooks have been published (see e.g. [Wri97, RTV97, Ye97]). Research is now concentrating on nonlinear optimization, especially on convex optimization.

Karmarkar’s algorithm was not competitive with the best simplex implementations, especially on small-scale problems, but his announcement concentrated a stream of research on the topic.

2The simplex method only shows an exponential complexity on some hand-crafted linear programs and is much faster on real-world problems, while the ellipsoid method always achieves its worst-case polynomial number of iterations, which turns out to be slower than the simplex method.

We also point out that Khachiyan’s method is not properly speaking the first polynomial algorithm for linear optimization, since Fiacco and McCormick’s method has been shown a posteriori to be polynomial by Anstreicher [Ans90].

1.2 Building blocks

In this section, we are going to review the different concepts needed to get a correct understanding of interior-point methods. We start with the very well studied notion of duality for linear optimization (see e.g. [Sch86]).

1.2.1 Duality

Let us state again the standard form of a linear program

min_{x∈ℝ^{n}} c^{T} x s.t. \{ \begin{matrix}Ax = b \\ x ≥ 0 \end{matrix} .  (LP)

Using the same data (viz. A, b and c) it is possible to describe another linear program

max_{y∈ℝ^{m}} b^{T} y s.t. \{ \begin{matrix}A^{T} y ≤ c \\ y is free \end{matrix} .  (LD’)

As we will see later, this program is closely related to (LP) and is called the dual of LP (which will be called primal program). It is readily seen that this program may also be written as

max_{y∈ℝ^{m},s∈ℝ^{n}} b^{T} y s.t. \{ \begin{matrix}A^{T} y + s = c \\ s ≥ 0 and y free \end{matrix} .  (LD)

This extra slack vector s will prove useful in simplifying our notation and we will therefore mainly use this formulation of the dual. We also define the dual feasible and strictly feasible sets D and D^{+} in a similar fashion to the sets P and P^{+}

D = {(y, s) | A^{T} y + s = c and s ≥ 0} ,
D^{+} = {(y, s) | A^{T} y + s = c and s > 0} .

From now on, we will assume that matrix A has full row rank, i.e. that its rows are linearly independent^{3}. Because of the equation A^{T} y + s = c, this implies a one-to-one correspondence between the y and s variables in the dual feasible set. In the following, we will thus refer to either (y, s), y or s as the dual variables.

We now state various important facts about duality:

3This is done without loss of generality: if a row of A is linearly dependent on some other rows, we have that the associated constraint is either redundant (and can be safely ignored) or impossible to satisfy (leading to an infeasible problem), depending on the value of the right-hand side vector b.

⋄ If x is feasible for (LP) and (y, s) for (LD), we have b^{T} y ≤ c^{T} x. This means that any feasible point of (LD) provides a lower bound for (LP) and that any feasible point of (LP) provides an upper bound for (LD). This is the weak duality property. The nonnegative quantity c^{T} x − b^{T} y is called the duality gap and is equal to x^{T} s.

⋄ x and (y, s) are optimal for (LP) and (LD) if and only if the duality gap is zero. This is the strong duality property. This implies that when both problems have optimal solutions, their objective values are equal. In that case, since x^{T} s = 0 and x ≥ 0, s ≥ 0, we have that all products x_{i}s_{i} must be zero, i.e. at least one of x_{i} and s_{i} is zero for each i (this is known as complementary slackness).

⋄ One of the following three situations occurs for problems (LP) and (LD)

a. Both problems have finite optimal solutions.

b. One problem is unbounded (i.e. its optimal value is infinite) and the other one is infeasible (i.e. its feasible set is empty). In fact, the weak duality property is easily seen to imply that the dual of an unbounded problem cannot have any feasible solution.

c. Both problems are infeasible.

This result is known as the fundamental theorem of duality.

Let us point out that it is possible to generalize most of these duality results to the class of convex optimization problems (see Chapter 3).

1.2.2 Optimality conditions

Karush-Kuhn-Tucker (KKT) conditions are necessary optimality conditions pertaining to nonlinear constrained optimization with a differentiable objective. Moreover, they are sufficient when the problem is convex, which is the case for linear optimization. For problem (LP) they lead to the following system

x is optimal for (LP) ⇔ ∃ (z, t) s.t. \{ \begin{matrix}Ax & = & b \\ A^{T} z + t & = & c \\ x_{i}t_{i} & = & 0 ∀i \\ x and t & ≥ & 0 \end{matrix} .  (KKT)

The second equation has exactly the same structure as the equality constraint for the dual problem (LD). Indeed, if we identify z with y and t with s we find

x is optimal for (LP) ⇔ ∃ (y, s) s.t. \{ \begin{matrix}Ax & = & b \\ A^{T} y + s & = & c \\ x_{i}s_{i} & = & 0 ∀i \\ x and s & ≥ & 0 \end{matrix} .

Finally, using the definitions of P and D and the fact that when u and v are nonnegative

u_{i}v_{i} = 0 ∀i ⇔ ∑_{i} u_{i}v_{i} = 0 ⇔ u^{T} v = 0

we have

x is optimal for (LP) ⇔ ∃ (y, s) s.t. \{ \begin{matrix}x & ∈ & P \\ (y, s) & ∈ & D \\ x^{T} s & = & 0 \end{matrix} .

This is in fact a confirmation of the strong duality theorem, revealing the deep connections between a problem and its dual: a necessary and sufficient condition for the optimality of a feasible primal solution is the existence of a feasible dual solution with zero duality gap (i.e. the same objective value).

Similarly, applying the KKT conditions to the dual problem would lead exactly to the same set of conditions, requiring the existence of a feasible primal solution with zero duality gap.

1.2.3 Newton’s method

The fact that finding the optimal solution of a linear program is completely equivalent to solving the KKT conditions may suggest the use of a general method designed to solve systems of nonlinear equations^{4}. The most popular of these methods is the Newton’s method, whose principle is described in the following paragraph.

Let F : ℝ^{n} ↦ ℝ^{n} be a differentiable nonlinear mapping. Newton’s method is an iterative process aiming to find an x ∈ ℝ^{n} such that F (x) = 0. For each iterate x_{k}, the method computes a first-order approximation to F around x_{k} and sets x_{k+1} to the zero of this linear approximation. Formally, if J is the Jacobian of F (assumed to be nonsingular), we have

F (x_{k} + Δx_{k}) ≈ F (x_{k}) + J(x_{k})Δx_{k}

and the Newton step Δx_{k} is chosen such that this linear approximation is equal to zero: we let thus x_{k+1} = x_{k} + Δx_{k} where^{5} Δx_{k} = −J(x_{k})^{−1}F (x_{k}). Convergence to a solution is guaranteed if the initial iterate x_{0} lies in a suitable neighbourhood of one of the zeros of F.

Newton’s method is also applicable to minimization problems in the following way: let g : ℝ^{n} ↦ ℝ be a function to minimize. We form a second-order approximation to g(x) around x_{k}, namely

g(x_{k} + Δx_{k}) ≈ g(x_{k}) + ∇g(x_{k})^{T} Δx_{k} + \frac{1}{2}Δx_{k}^{T} ∇^{2}g(x_{k})Δx_{k} .

If the Hessian ∇^{2}g(x_{k}) is positive definite, which happens when g is strictly convex, this approximation has a unique minimizer, which we take as next iterate. It is defined by Δx_{k} = −∇^{2}g(x_{k})^{−1}∇g(x_{k}), which leads to a method that is basically equivalent to applying Newton’s method to the gradient-based optimality condition ∇g(x) = 0.

One problem with the application of Newton’s method to the resolution of the KKT conditions is the nonnegativity constraints on x and s, which cannot directly be taken into account via the mapping F. One way of incorporating these constraints is to use a barrier term, as described in the next paragraph.

4Strictly speaking, the first two conditions are linear while only the x_{i}s_{i} = 0 equations are nonlinear. The nonnegativity constraints are not equations and cannot be handled by such a method.

5Computation of Δx_{k} is usually done with the linear system J(x_{k})Δx_{k} = −F (x_{k}) rather than computing explicitly J(x_{k})’s inverse.

1.2.4 Barrier function

A barrier function ϕ : ℝ^{+} ↦ ℝ is simply a differentiable function such that lim_{x→0^{+}} ϕ(x) = +∞. Using such a barrier, it is possible to derive a parameterized family of unconstrained problems from an inequality-constrained problem in the following way

min_{x∈ℝ^{n}} f(x) s.t. g_{i}(x) ≥ 0 ∀i  (G)
→ min_{x∈ℝ^{n}} f(x) + μ ∑_{i} ϕ(g_{i}(x)) ,  (G_{μ})

where μ ∈ ℝ^{+}. The purpose of the added barrier term is to drive the iterates generated by an unconstrained optimization method away from the infeasible zone (where one or more g_{i}’s are negative). Of course, we should not expect the optimal solutions to (G_{μ}) to be equal to those of (G). In fact each value of μ gives rise to a different problem (G_{μ}) with its own optimal solutions.

However, if we solve a sequence of problems (G_{μ}) with μ decreasing to zero, we might expect the sequence of optimal solutions we obtain to converge to the optimum of the original problem (G), since the impact of the barrier term is less and less significant compared to the real objective function. The advantage of this procedure is that each optimal solution in the sequence will satisfy the strict inequality constraints g_{i}(x) > 0, leading to a feasible optimal solution to (G)^{6}.

The application of this technique to linear optimization will lead to a fundamental notion in interior-point methods: the central path.

1.2.5 The central path

Interior-point researchers use the following barrier function, called the logarithmic barrier:

ϕ(x) = − log(x) .

Using ϕ, let us apply a barrier term to the linear optimization problem (LP)

min_{x∈ℝ^{n}} c^{T} x − μ ∑_{i} log(x_{i}) s.t. \{ \begin{matrix}Ax = b \\ x > 0 \end{matrix}  (P_{μ})

and to its dual (LD) (since it is a maximization problem, we have to subtract the barrier term)

max_{y∈ℝ^{m}} b^{T} y + μ ∑_{i} log(s_{i}) s.t. \{ \begin{matrix}A^{T} y + s = c \\ s > 0 and y free \end{matrix} .  (D_{μ})

6The notion of barrier function was first investigated in [Fri55, FM68].

It is possible to prove (see e.g. [RTV97]) that both of these problems have unique optimal solutions x_{μ} and (y_{μ}, s_{μ}) for all μ > 0 if and only if both P^{+} and D^{+} are nonempty^{7}. In that case, we call the sets of optimal solutions {x_{μ} | μ > 0} ⊂ P^{+} and {(y_{μ}, s_{μ}) | μ > 0} ⊂ D^{+} respectively the primal and dual central paths. These parametric curves have the following properties:

⋄ The primal (resp. dual) objective value c^{T} x (resp. b^{T} y) is monotonically decreasing (resp. increasing) along the primal (resp. dual) central path when μ → 0.

⋄ The duality gap c^{T} x_{μ} − b^{T} y_{μ} for the primal-dual solution (x_{μ}, y_{μ}, s_{μ}) is equal to nμ. For this reason, μ will be called the duality measure. When a point (x, y, s) does not lie exactly on the central path, we can compute its estimated duality measure using μ = (c^{T} x − b^{T} y)/n.

⋄ The limit points x_{∗} = lim_{μ→0} x_{μ} and (y_{∗}, s_{∗}) = lim_{μ→0}(y_{μ}, s_{μ}) exist and hence are optimal solutions to problems (LP) and (LD) (because we have c^{T} x_{∗} − b^{T} y_{∗} = 0). Moreover, we have that x_{∗} + s_{∗} > 0, i.e. this optimal pair is strictly complementary^{8}.

1.2.6 Link between central path and KKT equations

To conclude this section we establish a link between the central path and the KKT equations. Applying the general KKT conditions to either problem (P_{μ}) or (D_{μ}) we find the following necessary and sufficient conditions

\{ \begin{matrix}Ax & = & b \\ A^{T} y + s & = & c \\ x_{i}s_{i} & = & μ ∀i \\ x and s & > & 0 \end{matrix} ⇔ \{ \begin{matrix}x & ∈ & P^{+} \\ (y, s) & ∈ & D^{+} \\ x_{i}s_{i} & = & μ ∀i \end{matrix} .  (KKT_{μ})

This system is very similar to the original KKT system, the only difference being the right-hand side of the third condition and the strict inequalities. This means in fact that the points on the central path satisfy a slightly perturbed version of the optimality KKT conditions for (LP) and (LD).

We now have all the tools we need to give a description of interior-point methods for linear optimization.

1.3 Interior-point algorithms

Since Karmarkar’s breakthrough, many different interior-point methods have been developed. It is important to note that there exists in fact a whole collection of methods, sharing the same basic principles but whose individual characteristics may vary a lot.

7This condition is known as the interior-point condition.

8For optimal solutions (x, s) we always have x_{i}s_{i} = 0, i.e. at least one of x_{i} and s_{i} is zero. In the case of a strictly complementary solution, exactly one of x_{i} and s_{i} is zero.

Among the criteria that are commonly used to classify the methods, we have

⋄ Iterate space. A method is said to be primal, dual or primal-dual when its iterates belong respectively to the primal space, the dual space or the Cartesian product of these spaces.

⋄ Type of iterate. A method is said to be feasible when its iterates are feasible, i.e. satisfy both the equality and nonnegativity constraints. In the case of an infeasible method, the iterates need not satisfy the equality constraints, but are still required to satisfy the nonnegativity conditions.

⋄ Type of algorithm. This is the main difference between the methods. Although the denominations are not yet fully standardized, we will distinguish path-following algorithms, affine-scaling algorithms and potential reduction algorithms. Sections 1.3.1, 1.3.2 and 1.3.3 will describe these three types of algorithms with more detail.

⋄ Type of step. In order to preserve their polynomial complexity, some algorithms are obliged to take very small steps at each iteration, leading to a high total number of iterations when applied to practical problems^{9}. These methods are called short-step methods and are mainly of theoretical interest. Therefore long-step methods, which are allowed to take much longer steps, have been developed and are the only methods used in practice.

It is not our purpose to give an exhaustive list of all the methods that have been developed up to now, but rather to present some representative algorithms, highlighting their underlying principles.

1.3.1 Path-following algorithms

We start with the most elegant category of methods, the path-following algorithms. As suggested by their denomination, the main idea behind these methods is to follow the central path up to its limit point. One could imagine the following naive conceptual algorithm (at this point, we want to keep generality and do not specify whether our method is primal, dual or primal-dual)

Given an initial iterate v_{0} and a sequence of duality measures monotonically decreasing to zero: μ_{1} > μ_{2} > μ_{3} > … > 0 and lim_{k→0} μ_{k} = 0.

Repeat for k = 0, 1, 2, …

Using v_{k} as starting point, compute v_{k+1}, the point on the central path with a duality measure equal to μ_{k+1}.

End

9Please note that this is not in contradiction with the fact that this number of iterations is polynomially bounded by the size of the problem. This may simply mean that the polynomial coefficients are large.

It is clear from this scheme that v_{k} will tend to the limit point of the central path, which is an optimal solution to our problem.

However, the determination of a point on the central path requires the solution of a minimization problem like (P_{μ}) or the (KKT_{μ}) conditions, which potentially implies a lot of computational work. This is why path-following interior-point methods only try to compute points that are approximately on the central path, hopefully with much less computational work, and will thus only loosely follow the central path. Our conceptual algorithm becomes

Given an initial iterate v_{0} and a sequence of duality measures monotonically decreasing to zero: μ_{1} > μ_{2} > μ_{3} > … > 0 and lim_{k→0} μ_{k} = 0. Repeat for k = 0, 1, 2, … Using v_{k} as starting point, compute v_{k+1}, an approximation of the point on the central path with a duality measure equal to μ_{k+1}. End

The main task in proving the convergence and complexity of these methods will be to assess how well we approximate our targets on the central path (i.e. how close to the central path we stay).

Short-step primal-dual path-following algorithm

This specific algorithm is a primal-dual feasible method, which means that all the iterates lie in P^{+} × D^{+}. Let (x_{k}, y_{k}, s_{k}) be the current iterate with duality measure μ_{k}. We also suppose that this iterate is close to the point (x_{μ_{k}}, y_{μ_{k}}, s_{μ_{k}}) on the central path. To compute the next iterate, we target (x_{μ_{k+1}}, y_{μ_{k+1}}, s_{μ_{k+1}}), a point on the central path with a smaller duality measure μ_{k+1} (thus closer to the optimal limit point). The main two characteristics of the short-step method are

⋄ The duality measure of the point we target is defined by μ_{k+1} = σμ_{k} where σ is a constant strictly between 0 and 1.

⋄ The next iterate will be computed by applying one single Newton step to the perturbed primal-dual conditions (KKT_{σμ_{k}}) defining our target on the central path^{10}

\{ \begin{matrix}Ax & = & b \\ A^{T} y + s & = & c \\ x_{i}s_{i} & = & σμ_{k} ∀i \end{matrix} .  (1.3)

Formally, we have presented Newton’s method as a way to find a root of a function F and not as a way to solve a systems of equations, so that we have first to define a function whose roots are solution of the system (1.3). Indeed, considering

F_{k} : ℝ^{2n+m} ↦ ℝ^{2n+m} : (\begin{matrix}x_{k} \\ y_{k} \\ s_{k} \end{matrix}) ↦ (\begin{matrix}Ax_{k} − b \\ A^{T} y_{k} + s_{k} − c \\ X_{k}S_{k}e − σμ_{k}e \end{matrix}) ,

10Note that we have to ignore the nonnegativity conditions for the moment.

where e stands for the all-one vector and X_{k} and S_{k} are diagonal matrices made up with vectors x_{k} and s_{k} (these notations are standard in the field of interior-point methods), we find that the Newton step we take is defined by the following linear system

(\begin{matrix}0 & A^{T} & I \\ A & 0 & 0 \\ S_{k} & 0 & X_{k} \end{matrix}) (\begin{matrix}Δx_{k} \\ Δy_{k} \\ Δs_{k} \end{matrix}) = (\begin{matrix}0 \\ 0 \\ −X_{k}S_{k}e + σμ_{k}e \end{matrix}) .  (1.4)

This leads to the following algorithm

Given an initial iterate (x_{0}, y_{0}, s_{0}) ∈ P^{+} × D^{+} with duality measure μ_{0} and a constant 0 < σ < 1.

Repeat for k = 0, 1, 2, …

Compute the Newton step (Δx_{k}, Δy_{k}, Δs_{k}) using the linear system (1.4).

Let (x_{k+1}, y_{k+1}, s_{k+1}) = (x_{k}, y_{k}, s_{k}) + (Δx_{k}, Δy_{k}, Δs_{k}) and μ_{k+1} = σμ_{k}.

End

We now sketch a proof of the correctness of this algorithm. For our path-following strategy to work, we have to ensure that our iterates (x_{k}, y_{k}, s_{k}) stay close to the points (x_{μ_{k}}, y_{μ_{k}}, s_{μ_{k}}) on the central path, which guide us to an optimal solution. For this purpose we define a quantity that measures the proximity between a strictly feasible iterate (x, y, s) ∈ P^{+} × D^{+} and the central point (x_{μ}, y_{μ}, s_{μ}). Since the main property of this central point is x_{i}s_{i} = μ ∀i, which is equivalent to^{11} xs = μe, the following measure (see e.g. [Wri97])

δ(x, s, μ) = \frac{1}{μ} ∥xs − μe∥ = ∥\frac{xs}{μ} − e∥

seems adequate: it is zero if and only if (x, y, s) is equal to (x_{μ}, y_{μ}, s_{μ}) and increases as we move away from this central point. It is also interesting to note that the size of a neighbourhood defined by δ(x, s, μ) < R decreases with μ, because of the leading term \frac{1}{μ}.

Another possibility of proximity measure with the same properties is

δ(x, s, μ) = \frac{1}{2} ∥\sqrt{\frac{xs}{μ}} − \sqrt{\frac{μ}{xs}}∥

where the square roots are taken componentwise (see [RTV97]).

The proof has the following steps [RTV97, Wri97]

a. Strict Feasibility. Prove that strict feasibility is preserved by the Newton step: if (x_{k}, y_{k}, s_{k}) ∈ P^{+} ×D^{+}, we have (x_{k+1}, y_{k+1}, s_{k+1}) ∈ P^{+} ×D^{+}. We have to be especially careful with the strict nonnegativity constraints, since they are not taken into account by Newton’s method. b. Duality measure. Prove that the target duality measure is attained after the Newton

11xs denotes here the componentwise product of vectors x and s.

step: if (x_{k}, y_{k}, s_{k}) has a duality measure equal to μ_{k}, the next iterate (x_{k+1}, y_{k+1}, s_{k+1}) has a duality measure equal to σμ_{k}

c. Proximity. Prove that proximity to the central path targets is preserved: there is a constant τ such that if δ(x_{k}, s_{k}, μ_{k}) < τ, we have δ(x_{k+1}, s_{k+1}, μ_{k+1}) < τ after the Newton step.

Adding the additional initial assumption that δ(x_{0}, s_{0}, μ_{0}) < τ, this is enough to prove that the sequence of iterates will stay in a prescribed neighbourhood of the central path and will thus (approximately) converge to its limit point, which is a (strictly complementary) optimal solution. The last delicate question is to choose a suitable combination of constants σ and τ that allows us to prove the three statements above. For the first duality measure we presented the following values are acceptable (see [Wri97])

σ = 1 − \frac{0.4}{\sqrt{n}} and τ = 0.4 ,

where n stands for the size of vectors x and s as usual, while for the second measure we may choose (see [RTV97])

σ = 1 − \frac{1}{2\sqrt{n}} and τ = \frac{1}{\sqrt{2}} .

To conclude this description, we specify how the algorithm terminates. Given an accuracy parameter ε, we stop our computations when the duality gap falls below ε, which happens when nμ_{k} < ε. This guarantees that c^{T} x and b^{T} y approximate the true optimal objective value with an error smaller than ε. We now state this algorithm in its final form:

Given an initial iterate (x_{0}, y_{0}, s_{0}) ∈ P^{+} × D^{+} with duality measure μ_{0}, an accuracy parameter ε and suitable constants 0 < σ < 1 and τ such that δ(x_{0}, y_{0}, s_{0}) < τ.

Repeat for k = 0, 1, 2, …

Compute the Newton step (Δx_{k}, Δy_{k}, Δs_{k}) using the linear system (1.4).

Let (x_{k+1}, y_{k+1}, s_{k+1}) = (x_{k}, y_{k}, s_{k}) + (Δx_{k}, Δy_{k}, Δs_{k}) and μ_{k+1} = σμ_{k}.

Until nμ_{k+1} < ε

Moreover, it is also possible to prove that in both cases, the solution with ε accuracy will be reached after a number of iterations N such that

N = O (\sqrt{n} log \frac{nμ_{0}}{ε} ) .  (1.5)

This polynomial complexity bound on the number of iterations that varies like the square root of the problem size is the best attained so far for linear optimization.

However, it is important to note that values of σ presented above will always be in practice nearly equal to one, which means that the duality measures will decrease very slowly. Although its complexity is polynomial, this method requires a large number of iterations and is not very efficient from a practical point of view.

Dual short-step path-following methods

This second short-step method is very similar to the previous one but its iterates lie in the dual space D^{+}. We keep the general principle of following the dual central path and targeting points (y_{μ_{k}}, s_{μ_{k}}) on it but we have to make the following adjustments^{12}

⋄ We cannot deduce the Newton step from the (KKT_{μ}) conditions any more, since they involve both primal and dual variables. We apply instead a single minimizing Newton step to the (D_{μ}) barrier problem, which gives the following (n + m) × (n + m) linear system

(\begin{matrix}A^{T} & I \\ AS_{k}^{−2}A^{T} & 0 \end{matrix}) (\begin{matrix}Δy_{k} \\ Δs_{k} \end{matrix}) = (\begin{matrix}0 \\ \frac{b}{σμ_{k}} − AS_{k}^{−1}e \end{matrix}) .  (1.6)

⋄ We have to modify our measure of proximity: we now define δ(s, μ) with [RTV97]

δ(s, μ) = min_{x} \{δ(x, s, μ) | Ax = b\} = \frac{1}{μ} min_{x} \{∥xs − μe∥ | Ax = b\}

(we have that this measure is zero if and only if s = s_{μ}).

Our algorithm simply becomes

Given an initial iterate (y_{0}, s_{0}) ∈ D^{+} with duality measure μ_{0}, an accuracy parameter ε and suitable constants 0 < σ < 1 and τ such that δ(y_{0}, s_{0}) < τ.

Repeat for k = 0, 1, 2, …

Compute the Newton step (Δy_{k}, Δs_{k}) using the linear system (1.6).

Let (y_{k+1}, s_{k+1}) = (y_{k}, s_{k}) + (Δy_{k}, Δs_{k}) and μ_{k+1} = σμ_{k}.

Until nμ_{k+1} < ε

In this case we may for example choose

σ = 1 − \frac{1}{3\sqrt{n}} and τ = \frac{1}{\sqrt{2}} ,

which leads to the same complexity bound (1.5) for the total number of iterations.

Primal-dual long-step path-following methods

The long-step primal-dual method we are going to describe now is an attempt to overcome the main limitation of the short-step methods: their very small step size. As presented above, the fundamental reason for this slow progress is the value of σ that has to be chosen nearly equal to one in order to prove the polynomial complexity of the method.

12It is of course also possible to design a primal short-step path-following method in a completely similar fashion.

A simple idea to accelerate the method would simply be to decrease the duality measure more aggressively, i.e. still using μ_{k+1} = σμ_{k} but with a lower σ. However, this apparently small change breaks down the good properties we were able to prove for the short-step algorithms. Indeed, if our target on the central path is too far from our current iterate, we may have that

⋄ The Newton step computed by (1.4) is no longer feasible. The reason for that is easy to understand. Newton’s method is asked to solve the (KKT_{μ}) system, which is made of two linear equations and one mildly nonlinear equation. Because of this third equation, the linear system we solve is only an approximation of the real set of equations, and the further we are from the solution we target, the less accurate this approximation is. When our target is located too far away, the linear approximation becomes so bad that barrier term does not play its role and the Newton step jumps out of the feasible region by violating the nonnegativity constraints^{13} x > 0 and s > 0.

Since the iterates of an interior-point method must always satisfy the strict nonnegativity conditions, we have to take a so-called damped Newton step, i.e. reduce it with a factor α_{k} < 1 in order to make it stay within the strictly feasible region P^{+} × D^{+}:

(x_{k+1}, y_{k+1}, s_{k+1}) = (x_{k}, y_{k}, s_{k}) + α_{k}(Δx_{k}, Δy_{k}, Δs_{k}) .

⋄ This damping of the Newton step cancels the property that the duality measure we target is attained. It is indeed possible to show that the duality measure after a damped Newton step becomes (1−α_{k}(1−σ))μ_{k}, which varies linearly between μ_{k} and σμ_{k} when α decreases from 1 to 0.

There is unfortunately no way to circumvent this drawback, and we have to accept that our iterates never exactly achieve the targeted duality measures, unless a full Newton step is taken.

⋄ We cannot guarantee that a single Newton step will keep the proximity to the central path in the sense of δ(x, s, μ) < τ, for the same reasons as above (nonlinearity). In the long-step strategy we describe, we take several Newton steps with the same target duality measure until proximity to the central path is restored. Then we may choose another target and decrease μ.

Our long-step method may be described in the following way:

Given an initial iterate (x_{0}, y_{0}, s_{0}) ∈ P^{+} × D^{+}, an initial duality measure μ_{0}, an accuracy parameter ε and suitable constants 0 < σ < 1 and τ such that δ(x_{0}, y_{0}, s_{0}) < τ.

Repeat for k = 0, 1, 2, …

Compute the Newton step (Δx_{k}, Δy_{k}, Δs_{k}) using the linear system (1.4).

Let (x_{k+1}, y_{k+1}, s_{k+1}) = (x_{k}, y_{k}, s_{k}) + α_{k}(Δx_{k}, Δy_{k}, Δs_{k}) with a step length α_{k} chosen such that (x_{k+1}, y_{k+1}, s_{k+1}) ∈ P^{+} × D^{+}.

13Note that since the first two conditions Ax = b and A^{T} y + s = c are linear, they are always fulfilled after the Newton step.

If δ(x_{k+1}, s_{k+1}, σμ_{k}) < τ Then let μ_{k+1} = σμ_{k} Else let μ_{k+1} = μ_{k}.

Until nμ_{k+1} < ε

As opposed to the complexity analysis of the short-step method, we may choose here whatever value we want for the constant σ, in particular values much smaller than 1. It is the choice of τ and α_{k} that makes the method polynomial. The main task is here to analyse the number of iterations that is needed to restore proximity to the central path. Taking for σ a constant independent of n (like .5, .1 or .01), it is possible to prove that suitable choices of τ and α_{k} lead to the following number of iterations

N = O (n log \frac{nμ_{0}}{ε} ) .

Let us point out an odd fact: although this method takes longer steps and is practically more efficient than the short-step methods, its theoretical complexity is worse than the short-step complexity (1.5).

1.3.2 Affine-scaling algorithms

The intensive stream of research on the topic of interior-point methods for linear optimization was triggered by Karmarkar’s seminal article [Kar84]. His method used projective transformations and was not described in terms of central path or Newton’s method. Later, researchers simplified this algorithm, removing the need for projective transformations, and obtained a class of methods called affine-scaling algorithms. It was later discovered that these methods had been previously proposed by Dikin in Russia, 17 years before Karmarkar [Dik67].

Affine-scaling algorithms do not explicitly follow the central path and do not even refer to it. The basic idea underlying these methods is the following: consider for example the primal problem (LP)

min_{x∈ℝ^{n}} c^{T} x s.t. \{ \begin{matrix}Ax = b \\ x ≥ 0 \end{matrix} .  (LP)

This problem is hard to solve because of the nonnegativity constraints, which give the feasible region a polyhedral shape. Let us consider the current iterate x_{k} and replace the polyhedral feasible region by an inscribed ellipsoid centered at x_{k}. The idea is to minimize the objective on this ellipsoid, which should be easier than on a polyhedron, and take this minimum as next iterate.

How do we construct an ellipsoid that is centered at x_{k} and inscribed into the feasible region ? Consider a positive diagonal matrix D. It is easy to show that problem (P_{D})

min_{w∈ℝ^{n}}(Dc)^{T} w s.t. \{ \begin{matrix}ADw = b \\ w ≥ 0 \end{matrix}  (P_{D})

is equivalent to (LP), the x variable being simply scaled by x = Dw (this scaling operation is responsible for the denomination of the method). Choosing a special diagonal matrix D = X_{k}, which maps the current iterate x_{k} to e, we obtain the following problem

min_{w∈ℝ^{n}}(X_{k}c)^{T} w s.t. \{ \begin{matrix}AX_{k}w = b \\ w ≥ 0 \end{matrix} .

We are now able to restrict the feasible region defined by w ≥ 0 to a unit ball centered at e, according to the inclusion {w | ∥w − e∥ ≤ 1} ⊂ {w | w ≥ 0}. Our problem becomes

min_{w∈ℝ^{n}}(X_{k}c)^{T} w s.t. \{ \begin{matrix}AX_{k}w = b \\ ∥w − e∥ ≤ 1 \end{matrix} ,

i.e. the minimization of a linear objective over the intersection of a unit ball and an affine subspace, whose solution can be easily computed analytically via a linear system. Back in the original space, this is equivalent to

min_{x∈ℝ^{n}} c^{T} x s.t. \{ \begin{matrix}Ax = b \\ ∥X_{k}^{−1}x − e∥ ≤ 1 \end{matrix} ,

whose feasible region is an ellipsoid centered at x_{k}. This ellipsoid is called the Dikin ellipsoid and lies entirely inside P. The minimum over this ellipsoid is given by x_{k} + Δx_{k}, where^{14}

Δx_{k} = −\frac{X_{k}P_{AX_{k}}X_{k}c}{∥P_{AX_{k}}X_{k}c∥} .  (1.7)

Because our ellipsoid lies entirely within the feasible region, the step Δx_{k} is feasible and the next iterate x_{k} + Δx_{k} is expected to be closer to the optimal solution than x_{k}.

Short- and long-step primal affine-scaling algorithms

Introducing a constant ρ to reduce the step size, we may state our algorithm as

Given an initial iterate x_{0} ∈ P^{+} and a constant 0 < ρ < 1.

Repeat for k = 0, 1, 2, …

Compute the affine scaling step Δ_{k} with (1.7) and let x_{k+1} = x_{k} + ρΔ_{k}.

End

This scheme is known as the short-step primal affine-scaling algorithm. Convergence to a primal solution has been proved for ρ = \frac{1}{8}, but we still do not know whether this method has polynomial complexity^{15}. It is of course possible to design a dual and even a primal-dual variant of this method (all we have to do is to define the corresponding Dikin ellipsoids).

It is also possible to make the algorithm more efficient by taking longer steps, i.e. moving outside of the Dikin ellipsoid. Keeping the same direction as for the short-step method, the maximum step we can take without leaving the primal feasible region is given by

Δx_{k} = −\frac{X_{k}P_{AX_{k}}X_{k}c}{max [P_{AX_{k}}X_{k}c]} ,  (1.8)

where max[v] stands for the maximum component of vector v, which leads to the following algorithm:

Q^{T} (QQ^{T})^{−1}Q when Q has maximal rank.

14P_{Q} denotes the projection matrix onto Ker Q, the null space of Q, which can be written as P_{Q} = I −

15When certain nondegeneracy conditions hold, convergence has been proved for 0 < ρ < 1.

Given an initial iterate x_{0} and a constant 0 < λ < 1.

Repeat for k = 0, 1, 2, …

Compute the affine scaling step Δ_{k} with (1.8) and let x_{k+1} = x_{k} + λΔ_{k}.

End

The constant λ decides which fraction of the way to the boundary of the feasible region we move^{16}. Global convergence has been proved when 0 < λ ≤ 2/3 but a surprising counterexample has been found with λ = 0.999 (see [Mas93]). Finally, as for the short-step method, we do not know whether this method has polynomial complexity.

Link with path-following algorithms

There is an interesting and unexpected link between affine-scaling methods and path-following algorithms. Taking for example the definition (1.6) of the dual Newton step in the pathfollowing framework and letting σ tend to zero, i.e. letting the target duality measure tend to zero, we find that the resulting limit direction is exactly equal to the dual affine-scaling direction ! This surprising fact, which is also valid for their primal counterparts, gives us some insight about both methods:

⋄ The affine-scaling method can be seen as an application of Newton’s method that is targeting the limit point of the central path, i.e. that tries to jump directly to an optimal solution without following the central path.

⋄ Looking at (1.6), it is possible to decompose the dual Newton step into two parts:

Δx_{k} = \frac{1}{σμ_{k}} Δ^{a}x_{k} + Δ^{c}x_{k} ,

where

(\begin{matrix}A^{T} & I \\ AS_{k}^{−2}A^{T} & 0 \end{matrix}) (\begin{matrix}Δ^{a}y_{k} \\ Δ^{a}s_{k} \end{matrix}) = (\begin{matrix}0 \\ b \end{matrix}) and (\begin{matrix}A^{T} & I \\ AS_{k}^{−2}A^{T} & 0 \end{matrix}) (\begin{matrix}Δ^{c}y_{k} \\ Δ^{c}s_{k} \end{matrix}) = (\begin{matrix}0 \\ −AS_{k}^{−1}e \end{matrix}) .

– Δ^{a}x_{k} is called the affine-scaling component. It has the same direction as the affine-scaling method and is only seeking optimality.

– Δ^{c}x_{k} is called the centering component. It is targeting a point on the central path with the same duality measure as the current iterate, i.e. only tries to improve proximity to the central path.

It is possible to show that most interior-point methods follow in fact directions that are combinations of these two basic directions.

16This constant has to be strictly less than 1 since we want to stay in the interior of the feasible region.

1.3.3 Potential reduction algorithms

Instead of targeting a decreasing sequence of duality measures, the method of Karmarkar made use of a potential function to monitor the progress of its iterates. A potential function is a way to measure the worth of an iterate. Its main two properties are the following:

⋄ It should tend to −∞ if and only if the iterates tend to optimality.

⋄ It should tend to +∞ when the iterates tend to the boundary of the feasible region without tending to an optimal solution^{17}.

The main goal of a potential reduction algorithm is simply to reduce the potential function by a fixed amount δ at each step, hence its name. Convergence follows directly from the first property above.

Primal-dual potential reduction algorithm

We are going to describe the application of this strategy in the primal-dual case. The Tanabe-Todd-Ye primal-dual potential function is defined on the strictly feasible primal-dual space P^{+} × D^{+} by

Φ_{ρ}(x, s) = ρ log x^{T} s − ∑_{i} log x_{i}s_{i} ,

where ρ is a constant required to be greater than n. We may rewrite it as

Φ_{ρ}(x, s) = (ρ − n) log x^{T} s − ∑_{i} log \frac{x_{i}s_{i}}{x^{T} s/n} + n log n

and note the following

⋄ The first term makes the potential tend to −∞ when (x, s) tends to optimality, since we have then the duality gap x^{T} s tending to 0.

⋄ The second term measures centrality of the iterate. A perfectly centered iterate will have all its products x_{i}s_{i} equal to their average value x^{T} s/n, making the second term equal to zero. As soon these products become different, this term increases, and tends to +∞ if one of the products x_{i}s_{i} tends to zero without x^{T} s tending also to zero (which means exactly that we approach the boundary of the feasible region without tending to an optimal solution).

The search direction for this method is not new: it is the same as for the path-following algorithm, defined with a target duality measure nμ_{k}/ρ (i.e. with σ = n/ρ). However, in this case, μ_{k} will not follow a predefined decreasing sequence, but will have to be recomputed after each step (since this algorithm cannot guarantee that the duality measure targeted by the Newton step will be attained). The algorithm proceeds as follows:

17We cannot of course simply prevent the method from approaching the boundary of the feasible region, since our optimal solution lies on it.

Given an initial iterate (x_{0}, y_{0}, s_{0}) ∈ P^{+} × D^{+} with duality measure μ_{0} and a constant ρ > n. Define σ = n/ρ.

Repeat for k = 0, 1, 2, …

Compute the Newton step (Δx_{k}, Δy_{k}, Δs_{k}) using the linear system (1.4).

Let (x_{k+1}, y_{k+1}, s_{k+1}) = (x_{k}, y_{k}, s_{k}) + α_{k}(Δx_{k}, Δy_{k}, Δs_{k}) where α_{k} is defined by

α_{k} = arg min_{α} Φ_{ρ}(x_{k} + αΔx_{k}, s_{k} + αΔs_{k})
s.t. (x_{k}, y_{k}, s_{k}) + α(Δx_{k}, Δy_{k}, Δs_{k}) ∈ P^{+} × D^{+} .

Evaluate μ_{k+1} with (x_{k+1}^{T}s_{k+1})/n.

Until nμ_{k+1} < ε

The principle of this method is thus to minimize the potential function along the search direction at each iteration. The main task in analysing the complexity of this method is to prove that this step will provide at least a fixed reduction of Φ_{ρ} at each iteration. Using ρ = n + \sqrt{n}, it is possible to prove that Φ_{ρ}(x_{k+1}, s_{k+1}) ≤ Φ_{ρ}(x_{k}, s_{k}) − δ with δ = 0.16 (see e.g. [Ans96]), leading to a total number of iterations equal to

N = O (\sqrt{n} log \frac{nμ_{0}}{ε} ) ,

matching the best complexity results for the path-following methods.

It is in general too costly for a practical algorithm to minimize exactly the potential function along the search direction, since Φ_{ρ} is a highly nonlinear function. We may use instead one the following strategies

⋄ Define a quadratic approximation of Φ_{ρ} along the search direction and take its minimizer as next iterate.

⋄ Take a fixed percentage (e.g. 95%) of the maximum step along the search direction staying inside of the feasible region.

We note however that polynomial complexity is no longer guaranteed in these cases.

1.4 Enhancements

In the following, we present various enhancements that are needed to make the theoretical methods of the previous section work in practice.

1.4.1 Infeasible algorithms

All the algorithms we have described up to now are feasible methods, which means they need a strictly feasible iterate as starting point. However, such a point is not always available:

⋄ For some problems, a natural strictly feasible point is not directly available and finding one may be as difficult as solving the whole linear program.

⋄ Some problems have no strictly feasible points although they are perfectly valid and have finite optimal solutions. This situation happens in fact if and only if the optimal solution set is not bounded^{18}.

We can think of two different strategies to handle such cases: embed the problem into a larger one that admits a strictly feasible starting point (this will be developed in the next paragraph) or modify the algorithm to make it work with infeasible iterates. We are now going to give an overview of this second strategy.

We recall that the iterates of an infeasible method do not satisfy the equality constraints Ax = b and A^{T} y + s = c but are required to be nonnegative, i.e. x > 0 and s > 0. The main idea is simply to ask Newton’s method to make the iterates feasible. This amounts to a simple modification of the linear system (1.4), which becomes

(\begin{matrix}0 & A^{T} & I \\ A & 0 & 0 \\ S_{k} & 0 & X_{k} \end{matrix}) (\begin{matrix}Δx_{k} \\ Δy_{k} \\ Δs_{k} \end{matrix}) = (\begin{matrix}c − A^{T} y_{k} − s_{k} \\ b − Ax_{k} \\ −X_{k}S_{k}e + σμ_{k}e \end{matrix}) .  (1.9)

The only difference with the feasible system is the right-hand side vector, which now incorporates the primal and dual residuals b − Ax_{k} and c − (A^{T} y_{k} + s_{k}). Newton steps will try to reduce both the duality measure and the iterate infeasibility at the same time.

Infeasible variants of both path-following and potential reduction methods have been developed using this search direction. Without going into the details, let us point out that an additional constraint on the step has to be enforced to ensure that infeasibility is reduced at least at the same pace as the duality measure (to avoid ending with an ”optimal” solution that would be infeasible). The complexity results for these methods are the same as those of their feasible counterparts, although the analysis is generally much more involved.

1.4.2 Homogeneous self-dual embedding

As mentioned in the previous subsection, another way to handle infeasibility is to embed our problem into a larger linear program that admits a known feasible starting point. We choose a starting iterate (x_{0}, y_{0}, s_{0}) such that x_{0} > 0 and s_{0} > 0 and define the following quantities

b̂ = b − Ax_{0}
ĉ = c − A^{T} y_{0} − s_{0}
ĝ = b^{T} y_{0} − c^{T} x_{0} − 1
ĥ = x_{0}^{T} s_{0} + 1 .

18This is the case for example when a variable that is not bounded by the constraints is not present in the objective.

We consider the following problem, introduced in [YTM94]

\begin{aligned}min & & & & ĥ θ \\ s.t. & & Ax & −b τ & +b̂ θ & & & = & 0 \\ & −A^{T} y & & +c τ & −ĉθ & −s & & = & 0 \\ & b^{T} y & −c^{T} x & & −ĝ θ & & −κ & = & 0 \\ & −b̂^{T} y & +ĉ^{T} x & +ĝ τ & & & & = & −ĥ \\ & & x ≥ 0 & τ ≥ 0 & & s ≥ 0 & κ ≥ 0 \end{aligned} .  (HSD)

It is easy to see find a strictly feasible starting point for this problem. Indeed, one can check that (x, y, s, τ, κ, θ) = (x_{0}, y_{0}, s_{0}, 1, 1, 1) is a suitable choice. Without going into too many details, we give a brief description of the new variables involved in (HSD): τ is a homogenizing variable, θ is measuring infeasibility and κ refers to the duality gap in the original problem. We also point out that the first two equalities correspond to the feasibility constraints Ax = b and A^{T} y + s = c.

This program has the following interesting properties (see [YTM94]):

⋄ This program is homogeneous, i.e. its right-hand side is the zero vector (except for the last equality that is a homogenizing constraint).

⋄ This program is self-dual, i.e. its dual is identical to itself (this is due to the fact that the coefficient matrix is skew-symmetric).

⋄ The optimal value of (HSD) is 0 (i.e. θ_{∗} = 0).

⋄ Given a strictly complementary solution (x_{∗}, y_{∗}, s_{∗}, τ_{∗}, κ_{∗}, 0) to (HSD) we have either τ_{∗} > 0 or κ_{∗} > 0.

⋄ If τ_{∗} > 0 then (x_{∗}/τ_{∗}, y_{∗}/τ_{∗}, s_{∗}/τ_{∗}) is an optimal solution to our original problem.

⋄ If κ_{∗} > 0 then our original problem has no finite optimal solution. Moreover, we have in this case b^{T} y_{∗} − c^{T} x_{∗} > 0 and

– When b^{T} y_{∗} > 0, problem (LP) is infeasible.

– When −c^{T} x_{∗} > 0, problem (LD) is infeasible.

Since we know a strictly feasible starting point, we can apply a feasible path-following method to this problem that will converge to an optimal strictly complementary solution. Using the above-mentioned properties, it is then possible to compute an optimal solution to our original problem or detect its infeasibility.

This homogeneous self-dual program has roughly twice the size of our original linear program, which may be seen as a drawback. However, it is possible to take advantage of the self-duality property and use some algorithmic devices to solve this problem at nearly the same computational cost as the original program.

1.4.3 Theory versus implemented algorithms

We have already mentioned that a polynomial complexity result is not necessarily a guarantee of good practical behaviour. Short-step methods are definitely too slow because of the tiny reduction of the duality measure they allow. Long-step methods perform better but are still too slow. This is why practitioners have implemented various tricks to accelerate their practical behaviour. It is important to note that the complexity results we have mentioned so far do not apply to these modified methods, since they do not strictly follow the theory.

The infeasible primal-dual long-step path-following algorithm is by far the most commonly implemented interior-point method. The following tricks are usually added:

⋄ The theoretical long-step method takes several Newton steps targeting the same duality measure until proximity to the central path is restored. Practical algorithms ignore this and take only a single Newton step, like short-step methods.

⋄ Instead of choosing the step length recommended by the theory, practical implementations usually take a very large fraction of the maximum step that stays within the feasible region (common values are 99.5% or 99.9%). This modification works especially well with primal-dual methods.

⋄ The primal and dual steps are taken with different step lengths, i.e. we take

x_{k+1} = x_{k} + α^{P} Δx_{k} and (y_{k+1}, s_{k+1}) = (y_{k}, s_{k}) + α^{D}(Δy_{k}, Δs_{k}) .

These steps are chosen according to the previous trick, for example with (α^{P} , α^{D}) = 0.995 (α_{max}^{P}, α_{max}^{D}). This modification alone is responsible for a substantial decrease of the total number of iterations, but is not theoretically justified.

1.4.4 The Mehrotra predictor-corrector algorithm

The description of the methods from the previous section has underlined the fact that the constant σ, defining the target duality measure σμ_{k}, has a very important role in determining the algorithm efficiency:

⋄ Choosing σ nearly equal to 1 allows us to take a full Newton step, but this step is usually very short and does not make much progress towards the solution. However it has the advantage of increasing the proximity to the central path.

⋄ Choosing a smaller σ produces a larger Newton step making more progress towards optimality, but this step is generally infeasible and has to be damped. Moreover this kind of step usually tends to move the iterate away from the central path.

We understand that the best choice of σ may vary according to the current iterate: small if a far target is easy to attain and large otherwise. Mehrotra has designed a very efficient way to choose σ according to this principle: the predictor-corrector primal-dual infeasible algorithm [Meh92].

This algorithm first computes an affine-scaling predictor step (Δx_{k}^{a}, Δy_{k}^{a}, Δs_{k}^{a}), i.e. solves (1.9) with σ = 0, targeting directly the optimal limit point of the central path. The maximum feasible step lengths are then computed separately using

α_{k}^{a,P} = arg max {α ∈ [0, 1] | x_{k} + αΔx_{k}^{a} ≥ 0} ,
α_{k}^{a,D} = arg max {α ∈ [0, 1] | s_{k} + αΔs_{k}^{a} ≥ 0} .

Finally, the duality measure of the resulting iterate is evaluated with

μ_{k+1}^{a} = \frac{(x_{k} + α_{k}^{a,P} Δx_{k}^{a})^{T} (α_{s}^{a,D}Δs_{k}^{a})}{n} .

This quantity measures how easy it is to progress towards optimality: if it is much smaller than the current duality measure μ_{k}, we can choose a small σ and hope to make much progress, on the other hand if it is just a little smaller, we have to be more careful and choose σ closer to one, in order to increase proximity to the central path and be in a better position to achieve a large decrease of the duality measure on the next iteration. Mehrotra suggested the following heuristic, which has proved to be very efficient in practice

σ = (\frac{μ_{k+1}^{a}}{μ_{k}} )^{3} .

We now simply compute a corrector step (Δx_{k}^{c}, Δy_{k}^{c}, Δs_{k}^{c}) using this σ and take the maximum feasible step lengths separately in the primal and dual spaces.

However, this algorithm can be improved a little further using the following fact. After a full predictor step, the pairwise product x_{i}s_{i} is transformed into (x_{i} + Δx_{i}^{a})(s_{i} + Δs_{i}^{a}), which can be shown to be equal to Δx_{i}^{a}Δs_{i}^{a}. Since Newton’s method was trying to make x_{i}s_{i} equal to zero, this last product measures the error due to the nonlinearity of the equations we are trying to solve. The idea is simply to incorporate this error term in the computation of the corrector step, using the following modification to the right-hand side in (1.9)

(\begin{matrix}0 & A^{T} & I \\ A & 0 & 0 \\ S_{k} & 0 & X_{k} \end{matrix}) (\begin{matrix}Δx_{k} \\ Δy_{k} \\ Δs_{k} \end{matrix}) = (\begin{matrix}c − A^{T} y_{k} − s_{k} \\ b − Ax_{k} \\ −X_{k}S_{k}e − ΔX_{k}^{a}ΔS_{k}^{a}e + σμ_{k}e \end{matrix}) .  (1.10)

This strategy of computing a step taking into account the results of a first-order prediction gives rise to a second-order method. The complete algorithm follows:

Given an initial iterate (x_{0}, y_{0}, s_{0}) with duality measure μ_{0} such that x_{0} > 0 and s_{0} > 0, an accuracy parameter ε and a constant ρ < 1 (e.g. 0.995 or 0.999). Repeat for k = 0, 1, 2, … Compute the predictor Newton step (Δx_{k}^{a}, Δy_{k}^{a}, Δs_{k}^{a}) using the linear system (1.9) and σ = 0. Compute the maximal step lengths and the resulting duality measure with

α_{k}^{a,P} = arg max {α ∈ [0, 1] | x_{k} + αΔx_{k}^{a} ≥ 0} ,
α_{k}^{a,D} = arg max {α ∈ [0, 1] | s_{k} + αΔs_{k}^{a} ≥ 0} ,
μ_{k+1}^{a} = \frac{(x_{k} + α_{k}^{a,P} Δx_{k}^{a})^{T} (s_{k} + α_{k}^{a,D}Δs_{k}^{a})}{n} .

Compute the corrector Newton step (Δx_{k}^{c}, Δy_{k}^{c}, Δs_{k}^{c}) using the modified linear system (1.10) and σ = (μ_{k+1}^{a}/μ_{k})^{3}.

Compute the maximal step lengths with

α_{k}^{P} = arg max {α ∈ [0, 1] | x_{k} + αΔx_{k}^{c} ≥ 0} ,
α_{k}^{D} = arg max {α ∈ [0, 1] | s_{k} + αΔs_{k}^{c} ≥ 0} .

Let x_{k+1} = x_{k} + ρ α_{k}^{P} Δx_{k}^{c} and (y_{k+1}, s_{k+1}) = (y_{k}, s_{k}) + ρ α_{k}^{D}(Δy_{k}^{c}, Δs_{k}^{c}).

Evaluate μ_{k+1} with (x_{k+1}^{T}s_{k+1})/n.

Until nμ_{k+1} < ε

It is important to note that the predictor step is only used to compute σ and the righthand side of (1.10) and is not actually taken. This has a very important effect on the computational work, since the calculation of both the predictor and the corrector step is made with the same current iterate. This implies that the coefficient matrix in the linear systems (1.10) and (1.9) is the same, the only difference being the right-hand side vector. The resolution of the second system will then reuse the factorization of the coefficient matrix and will only need a computationally cheap additional backsubstitution. This property is responsible for the great efficiency of Mehrotra’s algorithm: a clever heuristic to decrease the duality measure using very little additional computational work.

1.5 Implementation

We mention here some important facts about the implementation of interior-point algorithms.

1.5.1 Linear algebra

It is important to realize that the resolution of the linear system defining the Newton step takes up most of the computing time in interior-point methods (some authors report 80–90% of the total CPU time). It should be therefore very carefully implemented. Equations (1.9) are not usually solved in this format: some pivoting is done, leading first to the following system (where we define D_{k}^{2} = S_{k}^{−1}X_{k})

(\begin{matrix}−D_{k}^{−2} & A^{T} \\ A & 0 \end{matrix} ) (\begin{matrix}Δx_{k} \\ Δy_{k} \end{matrix}) = (\begin{matrix}c − A^{T} y_{k} − σμ_{k}X_{k}^{−1}e) \\ b − Ax_{k} \end{matrix})  (1.11)
Δs_{k} = −s_{k} + σμ_{k}X_{k}^{−1}e − D_{k}^{−2}Δx_{k} ,  (1.12)

and then to this one

AD_{k}^{2}A^{T} Δy_{k} = b − A(x_{k} − D_{k}^{2}c + D_{k}^{2}A^{T} y_{k} + σμ_{k}S_{k}^{−1}e)  (1.13)
Δs_{k} = c − A^{T} y_{k} − s_{k} − A^{T} Δy_{k}  (1.14)
Δx_{k} = −x_{k} + σμ_{k}S_{k}^{−1}e − D_{k}^{2}Δs_{k} .  (1.15)

System (1.11) is called the augmented system and can be solved with a Bunch-Partlett factorization. However, the most usual way to compute the Newton step is to solve (1.13), called the normal equation, with a Cholevsky factorization, taking advantage of the fact that matrix AD_{k}^{2}A^{T} is positive definite (see [AGMX96] for a discussion). At this stage, it is important to note that most real-world problems have very few nonzero entries in matrix A. It is thus very important to exploit this sparsity in order to reduce both computing times and storage capacity requirements. More specifically, one should try to find a reordering of the rows and columns of matrix AD_{k}^{2}A^{T} that leads to the sparsest Cholevsky factor^{19}. This permutation has to be computed only once, since the sparsity pattern of matrix AD_{k}^{2}A^{T} is the same for all iterations.

On a side note, let us note that the complexity of solving this linear system is O (n^{3}) arithmetic iterations, which gives the best interior-point methods a total complexity of

O(n^{3.5} log \frac{nμ_{0}}{ε} )

arithmetic operations^{20}.

1.5.2 Preprocessing

In most cases, the linear program we want to solve is not formulated in the standard form (1.2). The first task for an interior-point solver is thus to convert it by adding variables and constraints

⋄ Inequality constraints can be transformed into equality constraints with a slack variable: f^{T} x ≥ b ⇔ f^{T} x − s = b with s ≥ 0.

⋄ A free variable can be split into two nonnegative variables: x = x^{+} − x^{−} with x^{+} ≥ 0 and x^{−} ≥ 0. However this procedure has some drawbacks^{21} and practical solvers usually include a modification of the algorithm to handle free variables directly.

⋄ Lower bounds l ≤ x are handled using a translation x = x′ + l with x′ ≥ 0.

⋄ Upper bounds x ≤ u could be handled using a slack variable, but practical solvers usually implement a variation of the standard form that takes these bounds directly into account.

After this initial conversion, it is not unusual that a series of simple transformations can greatly reduce the size of the problem

⋄ Zero lines and columns are either redundant (and thus may be removed) or make the problem infeasible.

19Because the problem of finding the optimal reordering is NP-hard, heuristics have been developed, e.g. the minimum degree and minimum local fill-in heuristics.

20A technique of partial updating of the coefficient matrix AD_{k}^{2}A^{T} in the normal equation can reduce this total complexity to O (n^{3}).

21 It makes for example the optimal solution set unbounded and the primal-dual strictly feasible set empty.

⋄ Equality constraints involving only one variable are removed and used to fix the value of this variable.

⋄ Equality constraints involving exactly two variables can be used to pivot out one the variables.

⋄ Two identical lines are either redundant (one of them may thus be removed) or inconsistent (and make the problem infeasible).

⋄ Some constraints may allow us to compute lower and upper bounds for some variables. These bounds can improve existing bounds, detect redundant constraints or diagnose an infeasible problem.

Every practical solver applies these rules (and some others) repeatedly before starting to solve the problem.

1.5.3 Starting point and stopping criteria

The problem of finding a suitable starting point has already been addressed by the homogeneous self-dual embedding technique and the infeasible methods. In both cases, any iterate satisfying x_{0} > 0 and s_{0} > 0 can be chosen as starting point. However, the actual performance of the algorithm can be greatly influenced by this choice.

Although there is no theoretical justification for it, the following heuristic is often used to find a starting point. We first solve

min_{x∈ℝ^{n}} c^{T} x + \frac{ω}{2} x^{T} x s.t. Ax = b and min_{(y,s)∈ℝ^{m}×ℝ^{n}} b^{T} y + \frac{ω}{2} s^{T} s s.t. A^{T} y + s = c .

These convex quadratic programs can be solved analytically at a cost comparable to a single interior-point iteration. The negative components of the optimal x and s are then replaced with a small positive constant to give x_{0} and (y_{0}, s_{0}).

As described earlier, the stopping criteria is usually a small predefined duality gap ε_{g}. In the case of an infeasible method, primal and dual infeasibility are also monitored and are required to fall below some predefined value ε_{i}. One can use for example the following formulas

\frac{∥Ax − b∥}{∥b∥ + 1} < ε_{i}, \frac{∥A^{T} y + s − c∥}{∥c∥ + 1} < ε_{i}, \frac{∥c^{T} x − b^{T} y∥}{∥c^{T} x∥ + 1} < ε_{g} .

The denominators are used to make these measures relative and the +1 constant to avoid division by zero. However, when dealing with an infeasible problem, infeasible methods tend to see their iterates diverging towards infinity. Practical solvers usually detect this behaviour and diagnose an infeasible problem.

1.6 Concluding remarks

The theory of interior-point methods for linear optimization is now well established ; several textbooks on the topic have been published (see e.g. [Wri97, RTV97, Ye97]). From a practical point of view, interior-point methods compete with the best simplex implementations, especially for large-scale problems.

However some unsatisfying issues remain, in particular the gap between theoretical and implemented algorithms. Another interesting point is the number of iterations that is practically observed, almost independent from the problem size or varying like log n or n^{1/4}, instead of the \sqrt{n} theoretical bound.

Research is now concentrating on the adaptation of these methods to the nonlinear framework. Let us mention the following directions:

⋄ Semidefinite optimization is a promising generalization of linear optimization in which the nonnegativity condition on a vector x ≥ 0 is replaced by the requirement that a symmetric matrix X is positive semidefinite. This kind of problem has numerous applications in various fields, e.g. combinatorial optimization (with the famous Goemans-Williamson bound on the quality of a semidefinite MAXCUT relaxation [GW95]), control, classification (see [Gli98b] and Appendix A), structural optimization, etc. (see [VB96] for more information). The methods we have presented here can be adapted to semidefinite optimization with relatively little effort and several practical algorithms are able to solve this kind of problem quite efficiently.

⋄ In their brilliant monograph [NN94], Nesterov and Nemirovski develop a complete theory of interior-point methods applicable to the whole class of convex optimization problems. They are able to prove polynomial complexity for several types of interior-point methods and relate their efficiency to the existence of a certain type of barrier depending on the problem structure, a so-called self-concordant barrier. This topic is further discussed in Chapter 2.

Self-concordant functions

This chapter provides a self-contained introduction to the theory of self-conc-ordant functions [NN94 ] and applies it to several classes of structured convex optimization problems. We describe the classical short-step interior-point method and optimize its parameters to provide its best possible iteration bound. We also discuss the necessity of introducing two parameters in the definition of self-concordancy, how they react to addition and scaling and which one is the best to fix. A lemma from [dJRT95 ] is improved and allows us to review several classes of structured convex optimization problems and evaluate their algorithmic complexity, using the self-concordancy of the associated logarithmic barriers.

2.1 Introduction

We start with a presentation of convex optimization.

2.1.1 Convex optimization

Convex optimization deals with the following problem

inf_{x∈ℝ^{n}} f_{0}(x) s.t. x ∈ C ,  (C)

where C ⊆ ℝ^{n} is a closed convex set and f_{0} : C ↦ ℝ is a convex function defined on C. Convexity of f_{0} and C plays a very important role in this problem, since it is responsible for the following two important properties [Roc70a, SW70]:

⋄ Any local optimum for (C) is also a global optimum, which implies that the objective value is equal for all local optima. Moreover, all these optima can be shown to form a convex set.

⋄ It is possible to use Lagrange duality to derive a dual problem strongly related to (C). Namely, this pair of problems satisfies a weak duality property (the objective value of any feasible solution for one of these problems provides a bound on the optimum objective value for the dual problem) and, under a Slater-type condition, a strong duality property (equality and attainment of the optimum objective values for the two problems). These properties are described with more detail in Section 3.2.

We first note that it can be assumed with any loss of generality that the objective function f_{0} is linear, so that we can define it as f_{0}(x) = c^{T} x using a vector c ∈ ℝ^{n}. Indeed, it is readily seen that problem (C) is equivalent to the following problem with a linear objective:

inf_{x∈ℝ^{n+1}} x_{n+1} s.t. x ∈ C̄ ,

where C̄ ⊆ ℝ^{n+1} is suitably defined as

C̄ = {(x, t) ∈ ℝ^{n+1} | x ∈ C and f(x) ≤ t} .

We will thus consider in the rest of this chapter the problem

inf_{x∈ℝ^{n}} c^{T} x s.t. x ∈ C .  (CL)

It is interesting to ask ourselves how one can specify the data of a problem cast in such a form, i.e. how one can describe its objective function and feasible set. While specifying the objective function is easily done by providing vector c, describing the feasible set C, which is responsible for the structure of problem (CL), can be done in several manners.

a. The traditional way to proceed in nonlinear optimization is to provide a list of convex constraints defining C, i.e.

C = {x ∈ ℝ^{n} | f_{i}(x) ≤ 0 ∀i ∈ I = {1, 2, …, m}} ,  (2.1)

where each of the m functions f_{i} : ℝ^{n} ↦ ℝ is convex. This guarantees the convexity of C, as an intersection of convex level sets.

b. An alternative approach consists in considering the domain of a convex function. More precisely, we require the interior of C to be equal to the domain of a convex function. Extending the real line ℝ with the quantity +∞, we introduce the convex function F : ℝ^{n} ↦ ℝ ∪ {+∞} and define C as the closure of its effective domain, i.e.

C = cl dom F = cl {x ∈ ℝ^{n} | F (x) < +∞} .

Most of the time, we will require in addition F to be a barrier function for the set C, according to the following definition.

Definition 2.1. A function F is a barrier function for the convex set C if and only if it satisfies the following assumptions:

(a) F is smooth (three times continuously differentiable for our purpose),

(b) F is strongly convex, i.e. ∇^{2}F is positive definite,

(c) F (x) tends to +∞ whenever x tends to ∂C, the boundary of C (this is the barrier property).

Note 2.1. We also note that it is often possible to provide a suitable barrier function F for a convex set C given by a functional description (2.1) using the logarithmic barrier [Fri55] defined as

F : ℝ^{n} ↦ ℝ : x ↦ F (x) = − ∑_{i∈I} log(−f_{i}(x)) ,

where we define log x = +∞ whenever x ≤ 0. We have indeed to check that F is strongly convex and is a barrier function for C, which is not always the case (for example, in the case of C = ℝ_{+}, taking f_{1}(x) = |x| − x does not lead to a strongly convex F while f_{1}(x) = −x^{x} leads to F (x) = −x log x, which does not possess the barrier property).

c. It may also be worthwhile to consider the special case where C can be described as the intersection of a convex cone C ⊆ ℝ^{n} and an affine subspace b + L (where L is a linear subspace)

C = C ∩ (b + L) = {x ∈ C | x − b ∈ L} .

The resulting class of problems is known as conic optimization, and can be easily shown to be equivalent to convex optimization [NN94] (in practice, subspace b + L would be defined with a set of linear equalities).

Special treatment for the linear constraints, i.e. their representation as an intersection with an affine subspace, can be justified by the fact that these constraints are easier to handle than general nonlinear constraints. In particular, let us mention that it is usually easy for algorithms to preserve feasibility with respect to these constraints, and that they cannot cause a nonzero duality gap, i.e. strong duality is valid without a Slater-type assumption for linear optimization. We will not need to use this approach in this chapter. It will nevertheless constitute the main tool used in the second part of this thesis, which focuses on the topic of duality (see Chapters 4–7).

2.1.2 Interior-point methods

Among the different types of algorithms that can be applied to solve problem (CL), the socalled interior-point methods have gained a lot of popularity in the last two decades. This is mainly due to the following facts:

⋄ it is not only possible to prove convergence of these methods to an optimal solution but also to give a polynomial bound on the number of arithmetic operations needed to reach a solution within a given accuracy, ⋄ these methods can be implemented and applied successfully to solve real-world prob-

lems, especially in the fields of linear (where they compare favourably with the simplex method), quadratic and semidefinite optimization.

A fundamental ingredient in the elaboration of these methods is the above-mentioned notion of barrier function F for the set C. Namely, let us consider the following parameterized family of unconstrained minimization problems:

inf_{x∈ℝ^{n}} \frac{c^{T} x}{μ} + F (x) ,  (CL_{μ})

where parameter μ belongs to ℝ_{++} and is called the barrier parameter. The constraint x ∈ C of the original problem (CL) has been replaced by a penalty term F (x) in the objective function, which tends to +∞ as x tends to the boundary of C and whose purpose is to avoid that the iterates leave the feasible set (see the classical monograph [FM68]). Assuming existence of a minimizer x(μ) for each of these problems (strong convexity of F ensures uniqueness of such a minimizer), we call the set {x(μ) | μ > 0} ⊆ C the central path for problem (CL).

It is intuitively clear that as μ tends to zero, the first term proportional to the original objective \frac{c^{T} x}{μ} becomes preponderant in the sum, which implies that the central path converges to a solution that is optimal for the original problem. The principle behind interior-point methods will thus be to follow this central path until an iterate that is sufficiently close to the optimum is found.

However, two questions remain pending: how do we compute x(μ) and how do we choose a suitable barrier F. The first question is readily answered: interior-point methods rely on Newton’s method to compute these minimizers, which leads us to a refined version of the second question: is it possible to choose a barrier function F such that Newton’s method is provably efficient in solving subproblems (CL_{μ}) and has an algorithmic complexity that can be estimated ? This crucial question is thoroughly answered by the remarkable theory of self-concordant functions, first developed by Nesterov and Nemirovski [NN94], which we will present in the next section.

2.1.3 Organization of the chapter

The purpose of this chapter is to give a self-contained introduction to the theory of selfconcordant functions and to apply it to several classes of structured convex optimization problems. Section 2.2 introduces a definition of self-concordant functions and presents several equivalent conditions. A short-step interior-point method using these functions is then presented along with an explanation of how the proof of polynomiality works. Our contribution at this stage is the computation of the best possible iteration bound for this method (Theorem 2.5).

Section 2.3 deals with the construction of self-concordant functions. Scaling and addition of self-concordant functions are considered, as well as a discussion on the utility of two parameters in the definition of self-concordancy and how to fix one of them in the best possible way. We then present an improved version of a lemma from [dJRT95] (Lemma 2.3). This lemma is the main tool used in Section 2.4, where we review several classes of structured convex optimization problems and prove self-concordancy of the corresponding logarithmic barriers, improving the complexity results found in [dJRT95]. We conclude in Section 2.5 with some comments.

2.2 Self-concordancy

We start this section with a definition of a self-concordant function.

2.2.1 Definitions

We first recall the following piece of notation: the first, second and third differentials of a function F : ℝ^{n} ↦ ℝ evaluated at the point x will be denoted by ∇F (x), ∇^{2}F (x) and ∇^{3}F (x). These are linear mappings, and we have indeed

∇F (x) : ℝ^{n} ↦ ℝ : h_{1} ↦ ∇F (x)[h_{1}]
∇^{2}F (x) : ℝ^{n} × ℝ^{n} ↦ ℝ : (h_{1}, h_{2}) ↦ ∇^{2}F (x)[h_{1}, h_{2}]
∇^{3}F (x) : ℝ^{n} × ℝ^{n} × ℝ^{n} ↦ ℝ : (h_{1}, h_{2}, h_{3}) ↦ ∇^{3}F (x)[h_{1}, h_{2}, h_{3}] .

Definition 2.2. A function F : C ↦ R is called (κ, ν)-self-concordant for the convex set C ⊆ ℝ^{n} if and only if F is a barrier function according to Definition 2.1 and the following two conditions hold for all x ∈ int C and h ∈ ℝ^{n}:

∇^{3}F (x)[h, h, h] ≤ 2κ (∇^{2}F (x)[h, h])^{\frac{3}{2}} ,  (2.2)
∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x) ≤ ν  (2.3)

(note that the square root in (2.2) is well defined since its argument ∇^{2}F (x)[h, h] is positive because of the requirement that F is convex).

This definition does not exactly match the original definition of a self-concordant barrier in [NN94], but merely corresponds to the notion of strongly non-degenerate κ^{−2}-self-concordant barrier functional with parameter ν, that is general enough for our purpose. Note 2.2. We would like to point out that no absolute value is needed in (2.2): while some authors usually require the apparently stronger condition

|∇^{3}F (x)[h, h, h]| ≤ 2κ (∇^{2}F (x)[h, h])^{\frac{3}{2}} ,  (2.4)

this is not needed since it suffices to notice that inequality (2.2) also has to hold in the direction opposite to h, which gives

∇^{3}F (x)[−h, −h, −h] ≤ 2κ (∇^{2}F (x)[−h, −h])^{\frac{3}{2}} ⇔ −∇^{3}F (x)[h, h, h] ≤ 2κ (∇^{2}F (x)[h, h])^{\frac{3}{2}}

(using the fact that the n^{th}-order differential is homogeneous with degree n), which combined with (2.2) gives condition (2.4).

It is possible to reformulate conditions (2.2) and (2.3) into several equivalent inequalities that may prove easier to handle in some cases. However, before we list them, we would like to make a few comments about the use of inner products in our setting, following the line of thought of Renegar’s monograph [Ren00].

It is indeed important to realize that the definitions of gradient and Hessian, i.e. firstorder and second-order differentials are in fact dependent from inner product that is being used. Nevertheless, in most texts, it is customary to use the dot product^{1} as standard inner product. This has the disadvantage to make all developments a priori dependent from the coordinate system. However, Renegar notices that it is possible to develop the theory of selfconcordant functions in a completely coordinate-free manner, i.e. independently of a reference inner product. The is due to the fact that the two principal objects in this theory are indeed independent from the coordinate system: the Newton step n(x) and the intrinsic inner product ⟨·, ·⟩_{x}. Given a barrier function F and a point x belonging to its domain, these two objects are defined according to:

n(x) = −(∇^{2}F (x))^{−1}∇F (x) and ⟨α, β⟩_{x} = ⟨α, ∇^{2}F (x)β⟩ .

It is also convenient to introduce the intrinsic norm ∥·∥_{x} based on the intrinsic inner product ⟨·, ·⟩_{x} according to the usual definition ∥a∥_{x} = \sqrt{⟨a, a⟩_{x}}.

Let x ∈ int C and h ∈ ℝ^{n} and let us introduce the one-dimensional function F_{x,h} : ℝ ↦ ℝ : t ↦ F (x+th), the restriction of F along the line {x + th | t ∈ ℝ}. We are now in position to state several reformulations of conditions (2.2) and (2.3), grouped in the following two theorems:

Theorem 2.1. The following four conditions are equivalent:

∇^{3}F (x)[h, h, h] ≤ 2κ (∇^{2}F (x)[h, h])^{\frac{3}{2}} for all x ∈ int C and h ∈ ℝ^{n}  (2.5a)
F′′′_{x,h}(0) ≤ 2κF′′_{x,h}(0)^{\frac{3}{2}} for all x ∈ int C and h ∈ ℝ^{n}  (2.5b)
F′′′_{x,h}(t) ≤ 2κF′′_{x,h}(t)^{\frac{3}{2}} for all x + th ∈ int C and h ∈ ℝ^{n}  (2.5c)
(−\frac{1}{\sqrt{F′′_{x,h}(t)}})′ ≤ κ for all x + th ∈ int C and h ∈ ℝ^{n} .  (2.5d)

Proof. Since F_{x,h}(t) = F (x + th), we can write

F′_{x,h}(t) = ∇F (x + th)[h], F′′_{x,h}(t) = ∇^{2}F (x + th)[h, h] and F′′′_{x,h}(t) = ∇^{3}F (x + th)[h, h, h] .

Condition (2.5b) is thus simply condition (2.5a) written differently. Moreover, condition (2.5c) is equivalent to condition (2.5b) written for x + th instead of x. Finally, we note that

(−\frac{1}{\sqrt{F′′_{x,h}(t)}})′ ≤ κ ⇔ \frac{1}{2}F′′_{x,h}(t)^{− \frac{3}{2}} F′′′_{x,h}(t) ≤ κ ⇔ F′′′_{x,h}(t) ≤ 2κF′′_{x,h}(t)^{\frac{3}{2}} ,

which shows that (2.5d) and (2.5c) are equivalent.

1The dot product of two vector x and y whose coordinates are (α_{1}, α_{2}, …, α_{n}) and (β_{1}, β_{2}, …, β_{n}) in a given coordinate system is equal to ∑_{i=1}^{n} α_{i}β_{i}.

Theorem 2.2. The following four conditions are equivalent:

∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x) ≤ ν for all x ∈ int C  (2.6a)
F′_{x,h}(0)^{2} ≤ νF′′_{x,h}(0) for all x ∈ int C and h ∈ ℝ^{n}  (2.6b)
F′_{x,h}(t)^{2} ≤ νF′′_{x,h}(t) for all x + th ∈ int C and h ∈ ℝ^{n} (2.6c)
(−\frac{1}{F′_{x,h}(t)})′ ≥ \frac{1}{ν} for all x + th ∈ int C and h ∈ ℝ^{n} .  (2.6d)

Proof. Proving these equivalences is a little more involved than for the previous theorem. We start by showing that condition (2.6b) implies condition (2.6a). We can write

∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x) = ∇F (x)[(∇^{2}F (x))^{−1}∇F (x)] = F′_{x,(∇^{2}F (x))^{−1}∇F (x)}(0)
≤ \sqrt{ν}\sqrt{F′′_{x,(∇^{2}F (x))^{−1}∇F (x)}(0)} using condition (2.6b)
= \sqrt{ν}\sqrt{∇^{2}F (x)[(∇^{2}F (x))^{−1}∇F (x), (∇^{2}F (x))^{−1}∇F (x)]}
= \sqrt{ν}\sqrt{∇F (x)^{T} (∇^{2}F (x))^{−1}∇^{2}F (x)(∇^{2}F (x))^{−1}∇F (x)}
= \sqrt{ν}\sqrt{∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x) ,}

which implies condition (2.6a). Considering now the reverse implication, we have

F′_{x,h}(0)^{2} = (∇F (x)[h])^{2} = (∇F (x)^{T} h)^{2}
= (∇F (x)^{T} (∇^{2}F (x))^{−1}∇^{2}F (x)h)^{2} = ⟨(∇^{2}F (x))^{−1}∇F (x), h⟩_{x}^{2}
≤ ∥(∇^{2}F (x))^{−1}∇F (x)∥_{x}^{2} ∥h∥_{x}^{2} (using the Cauchy-Schwarz inequality)
= (∇F (x)^{T} (∇^{2}F (x))^{−1}∇^{2}F (x)(∇^{2}F (x))^{−1}∇F (x)) (h^{T} ∇^{2}F (x)h)
≤ ν∇^{2}F (x)[h, h] using condition (2.6a)
= νF′′_{x,h}(0) .

Condition (2.6c) is condition (2.6b) written for x + th instead of x, and we finally note that

(−\frac{1}{F′_{x,h}(t)})′ ≥ \frac{1}{ν} ⇔ F′_{x,h}(t)^{−2}F′′_{x,h}(t) ≥ \frac{1}{ν} ⇔ νF′′_{x,h}(t) ≥ F′_{x,h}(t)^{2} ,

which shows that (2.6d) and (2.6c) are equivalent.

The first three reformulations for each condition are well-known and can be found for example in [NN94, Jar96, Ren00]. Conditions (2.5d) and (2.6d) are less commonly used and were independently presented in [Bri00].

2.2.2 Short-step method

As outlined in the introduction, interior-point methods for convex optimization rely on a barrier function and the associated central path to solve problem (CL). Ideally, we would like our iterates to be a sequence of points on the central path x(μ_{0}), x(μ_{1}), …, x(μ_{k}), … for a sequence of barrier parameters μ_{k} tending to zero (and thus x(μ_{k}) tending to an optimal solution).

We already mentioned that Newton’s method, applied to problems (CL_{μ}), will be the workhorse to compute those minimizers. However, it would be too costly to compute each of these points with high accuracy, so that interior-point methods require instead their iterates to lie in a prescribed neighbourhood of the central path and its exact minimizers.

Let x_{k}, the k^{th} iterate, be an approximation of x(μ_{k}). A good proximity measure would be ∥x_{k} − x(μ_{k})∥ or, to be independent from the coordinate system, ∥x_{k} − x(μ_{k})∥_{x_{k}}. However, these quantities involve the unknown central point x(μ_{k}), and is therefore difficult to work with. Nevertheless, another elegant proximity measure can be used for that purpose. Let us define n_{μ}(x) to be the Newton step trying to minimize the objective in problem (CL_{μ}), which is thus aiming at x(μ). Since this objective is equal to F_{μ}(x) = \frac{c^{T} x}{μ} + F (x), we have

n_{μ}(x) = −(∇^{2}F_{μ}(x))^{−1}∇F_{μ}(x) = −(∇^{2}F (x))^{−1}(\frac{c}{μ} + ∇F (x))
= −\frac{1}{μ}(∇^{2}F (x))^{−1}c + n(x) .  (2.7)

Let us now define δ(x, μ), a measure of the proximity of x to the central point x(μ), as the intrinsic norm of the newton step n_{μ}(x), i.e. δ(x, μ) = ∥n_{μ}(x)∥_{x}. This quantity is indeed a good candidate to measure how far x lies from the minimizer x(μ), since the Newton step at x targeting x(μ) is supposed to be a good approximation of x(μ) − x. The goal of a shortstep interior-point method will be to trace the central path approximately, ensuring that the proximity δ(x_{k}, μ_{k}) is kept below a predefined bound for each iterate.

We are now in position to sketch a short-step algorithm. Given a problem of type (CL), a barrier function F for C, an upper bound on the proximity measure τ > 0, a decrease parameter 0 < θ < 1 and an initial iterate x_{0} such that δ(x_{0}, μ_{0}) < τ, we set k ← 0 and perform the following main loop:

a. μ_{k+1} ← μ_{k}(1 − θ)

b. x_{k+1} ← x_{k} + n_{μ_{k+1}}(x_{k})

c. k ← k + 1

The key is to choose parameters τ and θ such that δ(x_{k}, μ_{k}) < τ implies δ(x_{k+1}, μ_{k+1}) < τ, so that proximity to the central path is preserved. This is the moment where the selfconcordancy of the barrier function F comes into play. Indeed, it is precisely this property that will guarantee that such a choice is always possible.

2.2.3 Optimal complexity

In order to relate the two proximities δ(x_{k}, μ_{k}) and δ(x_{k+1}, μ_{k+1}), it is useful to introduce an intermediate quantity δ(x_{k}, μ_{k+1}), the proximity from an iterate to its next target on the central path. We have the following two properties:

Theorem 2.3. Let F be a barrier function satisfying (2.3), x ∈ dom F and μ^{+} = (1 − θ)μ. We have

δ(x, μ^{+}) ≤ \frac{δ(x, μ) + θ\sqrt{ν}}{1 − θ} .

Proof. Using (2.7), we have

μ^{+}n_{μ^{+}}(x) − μ^{+}n(x) = −(∇^{2}F (x))^{−1}c = μn_{μ}(x) − μn(x)
(dividing by μ) ⇔ (1 − θ)n_{μ^{+}}(x) − (1 − θ)n(x) = n_{μ}(x) − n(x)
⇔ (1 − θ)n_{μ^{+}}(x) = n_{μ}(x) − θn(x)
⇒ (1 − θ) ∥n_{μ^{+}}(x)∥_{x} ≤ ∥n_{μ}(x)∥_{x} + θ ∥n(x)∥_{x}
⇒ (1 − θ)δ(x, μ^{+}) ≤ δ(x, μ) + θ\sqrt{ν} ,

which implies the desired inequality, where we used to derive the last implication the fact that

∥n(x)∥_{x} = \sqrt{⟨n(x), n(x)⟩_{x}} = \sqrt{∇F (x)^{T} (∇^{2}F (x))^{−1}∇^{2}F (x)(∇^{2}F (x))^{−1}∇F (x)}
= \sqrt{∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x)} ≤ \sqrt{ν} ,

because of condition (2.3).

Theorem 2.4. Let F be a barrier function satisfying (2.2) and x ∈ dom F. Let us suppose δ(x, μ) < \frac{1}{κ}. We have that x + n_{μ}(x) ∈ dom F and

δ(x + n_{μ}(x), μ) ≤ \frac{κδ(x, μ)^{2}}{(1 − κδ(x, μ))^{2}} .

This proof is more technical and is omitted here ; it can be found in [NN94, Jar96, Ren00].

Note 2.3. It is now clear why the self-concordancy property relies on two separate conditions: one of them is responsible for the control of the increase of the proximity measure when the target on the central path is updated (Theorem 2.3), while the other guarantees that the proximity to the target can be restored, i.e. sufficiently decreased when taking a Newton step (Theorem 2.4).

Assuming for the moment that τ and θ can be chosen such that the proximity to central path is preserved at each iteration, we see that the number of iterations needed to attain a certain value of the barrier parameter μ_{e} depends solely on the ratio \frac{μ_{0}}{μ_{e}} and the value of θ. Namely, since μ_{k} = (1 − θ)^{k}μ_{0}, it is readily seen that this number of iterations is equal to

⌈log_{(1−θ)} \frac{μ_{e}}{μ_{0}} ⌉ = ⌈\frac{1}{log(1 − θ)} log \frac{μ_{e}}{μ_{0}} ⌉ .  (2.8)

Given a (κ, ν)-self-concordant function, we are now going to find a suitable pair of parameters τ and θ . Moreover, we will optimize this choice of parameters, i.e. try to provide the greatest reduction for the parameter μ at each iteration, in other words maximize θ in order to get the lowest possible total iteration count. Letting δ = δ(x_{k}, μ_{k}), δ′ = δ(x_{k}, μ_{k+1}) and δ^{+} = δ(x_{k+1}, μ_{k+1}) and assuming δ ≤ τ, we have to satisfy δ^{+} ≤ τ with the greatest possible value for θ.

Let us assume first that δ′ < \frac{1}{κ}. Using Theorem 2.4, we find that

δ^{+} ≤ \frac{κδ′^{2}}{(1 − κδ′)^{2}}

and therefore require that

\frac{κδ′^{2}}{(1 − κδ′)^{2}} ≤ τ .

This is equivalent to

(\frac{κδ′}{1 − κδ′} )^{2} ≤ κτ ⇔ (\frac{1}{κδ′} − 1)^{2} ≥ \frac{1}{κτ} ⇔ \frac{1}{κδ′} ≥ 1 + \frac{1}{\sqrt{κτ}}

(this also shows that the assumption κδ′ < 1 we made in the beginning was valid). Using now Theorem 2.3, we know that

δ′ ≤ \frac{δ + θ\sqrt{ν}}{1 − θ} ⇒ δ′ ≤ \frac{τ + θ\sqrt{ν}}{1 − θ} ⇔ \frac{1}{κδ′} ≥ \frac{1 − θ}{κτ + θκ\sqrt{ν}}

and thus require that

\frac{1 − θ}{κτ + θκ\sqrt{ν}} ≥ 1 + \frac{1}{\sqrt{κτ}} .

Letting Γ = κ\sqrt{ν} and β = \sqrt{κτ} we have

\frac{1 − θ}{β^{2} + θΓ} ≥ 1 + \frac{1}{β} ⇔ 1 − θ ≥ (1 + \frac{1}{β})(β^{2} + Γθ) ⇔ 1 − β − β^{2} ≥ θ (1 + Γ + \frac{Γ}{β}) ,

which means finally that we have to choose θ such that

θ ≤ \frac{1 − β − β^{2}}{1 + Γ + \frac{Γ}{β}}  (2.9)

in order to guarantee δ^{+} ≤ τ. We are now in position to optimize the value of θ, i.e. find the value of β that maximizes this upper bound. However, this value is likely to depend on Γ (and thus on κ and ν) in a complex way. We are therefore going to work with the following slightly worse upper bound, which has the advantage of allowing the optimization of β independently of Γ (we use the fact that Γ = κ\sqrt{ν} ≥ 1, see [NN94])

θ ≤ \frac{1}{Γ}(\frac{1 − β − β^{2}}{2 + \frac{1}{β}} ) = \frac{f(β)}{Γ} ( ≤ \frac{1 − β − β^{2}}{1 + Γ + \frac{Γ}{β}} ) .

It is now straightforward to maximize f(β): computing the derivative shows there is a unique maximizer when β ≈ 0.273 (the exact value is the real root of 1 − 2β − 5β^{2} − 4β^{3}) and our upper bound in (2.9) becomes \frac{0.65}{1+4.66Γ}. Translating back into our original quantities τ, κ and ν we find that we can choose

τ = \frac{β^{2}}{κ} ≈ \frac{1}{13.42κ} and θ = \frac{1 − β − β^{2}}{1 + Γ + \frac{Γ}{β}} ≈ \frac{1}{1.53 + 7.15κ\sqrt{ν}} ,  (2.10)

which is the best result obtainable if we want β to be independent from κ and ν (more precisely, it essentially corresponds to the best result in the case where κ\sqrt{ν} = 1). This improves several results from the literature, e.g. θ = \frac{1}{9κ\sqrt{ν}} in [Jar96] and θ = \frac{1}{1+8κ\sqrt{ν}} in [Ren00].

Before we conclude this section with a global complexity result, let us say a few words about termination of the algorithm. The most practical stopping criteria is a small target value μ_{e} for the barrier parameter, which gives the iteration bound (2.8). Our final iterate x_{e} will thus satisfy δ(x_{e}, μ_{e}) ≤ τ, which tells us it is not too far from x(μ_{e}), itself not too far from the optimum since μ_{e} is small. Indeed, using again the self-concordancy property of F, it is possible to derive the following bound on the accuracy of the final objective c^{T} x_{e}, i.e. its deviation from the optimal objective c^{T} x^{∗}

c^{T} x_{e} − c^{T} x^{∗} ≤ \frac{μ_{e}}{1 − 3κτ} κ\sqrt{ν}  (2.11)

(proof of this fact is omitted here and can easily be obtained combining Theorems 2.2.5 and 2.3.3 in [Ren00]). We are now ready to state our final complexity result:

Theorem 2.5. Given a convex optimization problem (CL), a (κ, ν)-self-concordant barrier F for C and an initial iterate x_{0} such that δ(x_{0}, μ_{0}) < \frac{1}{13.42κ}, one can find a solution with accuracy ϵ in

⌈(1.03 + 7.15κ\sqrt{ν}) log \frac{1.29μ_{0}κ\sqrt{ν}}{ϵ} ⌉ iterations.

Proof. Using our optimal values for θ and τ from (2.10) and the bound on the objective accuracy in (2.11), we find that the stopping threshold on the barrier parameter μ_{e} must satisfy

\frac{μ_{e}}{1 − 3/13.42}κ\sqrt{ν} ≤ ϵ ⇔ 1.29μ_{e}κ\sqrt{ν} ≤ ϵ ⇔ μ_{e} ≤ \frac{ϵ}{1.29κ\sqrt{ν}} .

Plugging this value into (2.8) we find that the total number of iterations can be bounded by (omitting the rounding bracket for clarity)

\frac{1}{log(1 − θ)} log \frac{μ_{e}}{μ_{0}} ≤ \frac{1}{log(1 − θ)} log \frac{ϵ}{1.29μ_{0}κ\sqrt{ν}}
= −\frac{1}{log(1 − θ)} log \frac{1.29μ_{0}κ\sqrt{ν}}{ϵ}
≤ (\frac{1}{θ} − \frac{1}{2}) log \frac{1.29μ_{0}κ\sqrt{ν}}{ϵ}
= (1.03 + 7.15κ\sqrt{ν}) log \frac{1.29μ_{0}κ\sqrt{ν}}{ϵ} ,

as announced (the third line uses the inequality \frac{1}{log(1−θ)} ≥ \frac{1}{2} − \frac{1}{θ}, which can be easily derived using the Taylor series of log x around 1).

2.3 Proving self-concordancy

The previous section has made clear that the self-concordancy property of the barrier function F is essential to derive a polynomial bound on the number of iterations of the short-step method. Moreover, smaller values for parameters κ and ν imply a lower total complexity. The next question we may ask ourselves is how to find self-concordant barriers (ideally with low parameters).

2.3.1 Barrier calculus

An impressive result in [NN94] states that every convex set in ℝ^{n} admits a (K, n)-self-concordant barrier, where K is a universal constant (independent of n). However, the universal barrier they provide in their proof is defined as a volume integral over an n-dimensional convex body, and is therefore difficult to evaluate in practice, even for simple sets in lowdimensional spaces. Another potential problem with this approach is that evaluating this barrier (and/or its gradient and Hessian) might take a number of arithmetic operations that grows exponentially with n, which would lead to an exponential algorithmic complexity for the short-step method, despite the polynomial iteration bound.

Another approach to find self-concordant function is to combine basic self-concordant functions using operations that are known to preserve self-concordancy (this approach is called barrier calculus in [NN94]). We are now going to describe two of these self-concordancy preserving operations, positive scaling and addition, and examine how the associated parameters are affected in the process.

Let us start with positive scalar multiplication.

Theorem 2.6. Let F be a (κ, ν)-self-concordant barrier for C ⊆ ℝ^{n} and λ ∈ ℝ_{++} a positive scalar. Then (λF) is also a self-concordant barrier for C with parameters (\frac{κ}{\sqrt{λ}}, λν).

Proof. It is clear that (λF) is also a barrier function (i.e. smoothness, strong convexity and the barrier property are obviously preserved by scaling). Looking at the restrictions (λF)_{x,h} = λF_{x,h}, we also have that

(λF)′_{x,h} = λF′_{x,h}, (λF)′′_{x,h} = λF′′_{x,h} and (λF)′′′_{x,h} = λF′′′_{x,h} .

Since F is (κ, ν)-self-concordant, we have (using conditions (2.5b) and (2.6b) from Theorems 2.1 and 2.2)

F′′′_{x,h}(0) ≤ 2κF′′_{x,h}(0)^{\frac{3}{2}} and F′_{x,h}(0)^{2} ≤ νF′′_{x,h}(0) for all x ∈ int C and h ∈ ℝ^{n} .

This is equivalent to

λF′′′_{x,h}(0) ≤ 2\frac{κ}{\sqrt{λ}}(λF′′_{x,h}(0))^{\frac{3}{2}} and (λF′_{x,h}(0))^{2} ≤ λνλF′′_{x,h}(0) for all x ∈ int C and h ∈ ℝ^{n} ,

which is precisely stating that (λF) is (\frac{κ}{\sqrt{λ}}, λν)-self-concordant.

This theorem show that self-concordancy is preserved by positive scalar multiplication, but that parameters κ and ν are both modified. It is interesting to note that these parameters do not occur individually in the iteration bound of Theorem 2.5 but are rather always appearing together in the expression κ\sqrt{ν}. This quantity, which we will call the complexity value of the barrier, is solely responsible for the polynomial iteration bound. Looking at what happens to it when F is scaled by λ, we find that the scaled complexity value is equal to \frac{κ}{\sqrt{ν}} \sqrt{λν} = κ\sqrt{ν}, i.e. that the complexity value is invariant to scaling. This means in fine that scaling a self-concordant barrier does not influence the algorithmic complexity of the associated short-step method, a property than could reasonably be expected from the start.

Let us now examine what happens when two self-concordant barriers are added.

Theorem 2.7. Let F be a (κ_{1}, ν_{1})-self-concordant barrier for C_{1} ⊆ ℝ^{n} and G be a (κ_{2}, ν_{2})- self-concordant barrier for C_{2} ⊆ ℝ^{n}. Then (F + G) is a self-concordant barrier for C_{1} ∩ C_{2} (provided this intersection is nonempty) with parameters (max{κ_{1}, κ_{2}}, λ_{1} + λ_{2}).

Proof. It is straightforward to see that (F + G) is a barrier function for C_{1} ∩ C_{2}. Looking at the restrictions (F + G)_{x,h}, we also have that

(F + G)′_{x,h} = F′_{x,h} + G′_{x,h}, (F + G)′′_{x,h} = F′′_{x,h} + G′′_{x,h} and (F + G)′′′_{x,h} = F′′′_{x,h} + G′′′_{x,h} .

We can write thus

(F + G)′′′_{x,h} = F′′′_{x,h} + G′′′_{x,h} ≤ 2κ_{1}F′′_{x,h}^{\frac{3}{2}} + 2κ_{2}G′′_{x,h}^{\frac{3}{2}}
≤ 2 max\{κ_{1}, κ_{2}\}(F′′_{x,h}^{\frac{3}{2}} + G′′_{x,h}^{\frac{3}{2}} )
≤ 2 max\{κ_{1}, κ_{2}\}(F′′_{x,h} + G′′_{x,h})^{\frac{3}{2}} = 2 max\{κ_{1}, κ_{2}\}(F + G)′′_{x,h}

(where we used for the third inequality the easily proven fact x^{\frac{3}{2}} +y^{\frac{3}{2}} ≤ (x+y)^{\frac{3}{2}} for x, y ∈ ℝ_{++}) and

|(F + G)′_{x,h}| = |F′_{x,h} + G′_{x,h}| ≤ |F′_{x,h}| + |G′_{x,h}|
≤ \sqrt{ν_{1}}\sqrt{F′′_{x,h}} + \sqrt{ν_{2}}\sqrt{G′′_{x,h}}
≤ \sqrt{ν_{1} + ν_{2}}\sqrt{F′′_{x,h} + G′′_{x,h}} = \sqrt{ν_{1} + ν_{2}}\sqrt{(F + G)′′_{x,h}}

(where we used for the third inequality the Cauchy-Schwarz inequality applied to vectors (\sqrt{ν_{1}}, \sqrt{ν_{2}}) and (\sqrt{F′′_{x,h}}, \sqrt{G′′_{x,h}})), which is precisely stating that (F +G) is (max\{κ_{1}, κ_{2}\}, ν_{1} + ν_{2})-self-concordant.

2.3.2 Fixing a parameter

As mentioned above, scaling a barrier function with a positive scalar does not affect its selfconcordancy, i.e. its suitability as a tool for convex optimization, and leaves its complexity value unchanged. One can thus make the decision to fix one of the two parameters κ and ν arbitrarily and only work with the corresponding subclass of barrier, without any real loss of generality. We describe now two choices of this kind that have been made in the literature.

First choice. Some authors [dJRT95, RT98, Jar89, dRT92] choose to work with the second parameter ν fixed to one. However, this choice is not made explicitly but results from the particular structure of the barrier functions that are considered. Indeed, these authors consider convex optimization problems whose feasible sets are given by a functional description like (2.1), i.e.

inf_{x∈ℝ^{n}} c^{T} x s.t. f_{i}(x) ≤ 0 ∀i ∈ I .

In order to apply the interior-point method methodology, a barrier function is needed and it is customary to use the logarithmic barrier as described in Note 2.1

F : ℝ^{n} ↦ ℝ : x ↦ F (x) = − ∑_{i∈I} log(−f_{i}(x)) .

The following lemma will prove useful.

Lemma 2.1. Let f : ℝ^{n} ↦ ℝ be a convex function and define F : ℝ^{n} ↦ ℝ ∪ {+∞} : x ↦ − log(−f(x)), whose effective domain is the set C = {x ∈ ℝ^{n} | f(x) < 0}. We have that F satisfies the second condition of self-concordancy (2.3) with parameter ν = 1.

Proof. Using the equivalent condition (2.6b) of Theorem 2.2, we have to evaluate for x ∈ int C, h ∈ ℝ^{n} and t = 0

F′_{x,h}(t) = −\frac{∇f(x + th)[h]}{f(x + th)} and F′′_{x,h}(t) = \frac{∇f(x + th)[h]^{2} − ∇^{2}f(x + th)[h, h]f(x + th)}{f(x + th)^{2}} ,

which implies

F′_{x,h}(0)^{2} = \frac{∇f(x)[h]^{2}}{f(x)^{2}} ≤ \frac{∇f(x)[h]^{2} − ∇^{2}f(x)[h, h]f(x)}{f(x)^{2}} = F′′_{x,h}(0)

(where we have used the fact that ∇^{2}f(x)[h, h] ≥ 0 because f is convex and f(x) ≤ 0 because x belongs to the feasible set C), which implies that F satisfies the second self-concordancy condition (2.3) with ν = 1.

Since the complete logarithmic barrier is a sum of terms for which this lemma is applicable, we can use Theorem 2.7 to find that it satisfies the same condition with ν = |I| = m, the number of constraints.

This means that we only have to check the first condition (2.2) involving κ to establish self-concordancy for the logarithmic barrier. Assuming that each individual term − log(−f_{i}(x)) can be shown to satisfy it with κ = κ_{i}, we have that the whole logarithmic barrier is (max_{i∈I}\{κ_{i}\}, m)-self-concordant, which leads to a complexity value equal to ∥κ∥_{∞} \sqrt{m}, where we have defined κ = (κ_{1}, κ_{2}, …, κ_{m}).

Second choice. Another arbitrary choice of self-concordance parameters that one encounters frequently in the literature consists in fixing κ = 1 in the first self-concordancy condition (2.2). This approach has been used increasingly in the recent years (see e.g.

[NN94, Ren00, Jar96]), and we propose to give here a justification of its superiority over the alternative presented above.

Let us consider the same logarithmic barrier, and suppose again that each individual term F_{i} : x ↦ − log(−f_{i}(x)) has been shown to satisfy the first self-concordancy condition (2.2) with κ = κ_{i}. Our previous discussion implies thus that F_{i} is (κ, 1)-self-concordant. Multiplying now F_{i} with κ_{i}^{2}, Theorem 2.6 implies that κ_{i}^{2}F_{i} is (1, κ_{i}^{2})-self-concordant. The corresponding complete scaled logarithmic barrier

F̃ : x ↦ − ∑_{i∈I} κ_{i}^{2} log(−f_{i}(x))

is then (1, ∑_{i∈I} κ_{i}^{2})-self-concordant by virtue of Theorem 2.7, which leads finally to a com-

plexity value equal to \sqrt{∑_{i∈I} κ_{i}^{2}} = ∥κ∥_{2}. This quantity is always lower than the complexity value for the standard logarithmic barrier considered above because of the well-known norm inequality ∥κ∥_{2} ≤ \sqrt{m} ∥κ∥_{∞}, which proves the superiority of this second approach (the only case where they are equivalent is when all parameters κ_{i}’s are equal).

Note 2.4. The fundamental reason why the first approach is less efficient is that it makes us combine barriers with different κ parameters, with the consequence that only the largest value max_{i∈I}{κ_{i}} appears in the final complexity value (the other smaller values become completely irrelevant and do not influence the final complexity at all). The second approach avoids this situation by ensuring that κ is always equal to one, which means that κ’s are equal for each combination and that the final complexity is well depending on the parameters of all the terms of the logarithmic barrier.

2.3.3 Two useful lemmas

We have seen so far how to construct self-concordant barrier by combining simpler functionals but still have no tool to prove self-concordancy of these basic barriers. The purpose of this section is to present two lemmas that can help us in that regard.

The first one deals with the second condition of self-concordancy with logarithmically homogeneous barriers [NN94].

Lemma 2.2. Let us suppose F is a logarithmically homogeneous function with parameter α, i.e.

F (tx) = F (x) − α log t .  (2.12)

We have that F satisfies the second condition of self-concordancy (2.3) with parameter ν = α.

Proof. This fact admits the following straightforward proof. We start by differentiating both sides of (2.12) with respect to t, to find

∇F (tx)[x] = −\frac{α}{t} .

Fixing t = 1 gives

∇F (x)[x] = ∇F (x)^{T} x = −α .  (2.13)

Differentiating this last equality again, this time with respect to x, leads to

∇F (x) + ∇^{2}F (x)x = 0 ⇔ ∇F (x) = −∇^{2}F (x)x .  (2.14)

Looking now at the left-hand side in (2.3) we have

∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x) = −∇F (x)^{T} (∇^{2}F (x))^{−1}∇^{2}F (x)x = −∇F (x)^{T} x = α

(using successively (2.14) and (2.13)), which implies immediately that F satisfies the second condition of self-concordancy (2.3) with ν = α. It is worth to point out that this inequality is in this case always satisfied with equality.

The second lemma we are going to present deals with the first self-concordancy condition. Let us first introduce two auxiliary functions r_{1} and r_{2}, whose graphs are depicted in Figure 2.1:

r_{1} : ℝ ↦ ℝ : γ ↦ max\{1, \frac{γ}{\sqrt{3 − 2/γ}} \} and r_{2} : ℝ ↦ ℝ : γ ↦ max\{1, \frac{γ + 1 + 1/γ}{\sqrt{3 + 4/γ + 2/γ^{2}}} \} .

Both of these functions are equal to 1 for γ ≤ 1 and strictly increasing for γ ≥ 1, with the asymptotic approximations r_{1}(γ) ≈ \frac{γ}{\sqrt{3}} and r_{2}(γ) ≈ \frac{γ+1}{\sqrt{3}} when γ tends to +∞.

Figure 2.1: Graphs of functions r_{1} and r_{2}

Lemma 2.3. Let us suppose F is a convex function with effective domain C ⊆ ℝ_{+}^{n} and that there exists a constant γ such that

∇^{3}F (x)[h, h, h] ≤ 3γ∇^{2}F (x)[h, h]\sqrt{∑_{i=1}^{n} \frac{h_{i}^{2}}{x_{i}^{2}}} for all x ∈ int C and h ∈ ℝ^{n} .  (2.15)

We have that

F_{1} : C ↦ ℝ : x ↦ F (x) − ∑_{i=1}^{n} log x_{i}

satisfies the first condition of self-concordancy (2.2) with parameter κ_{1} = r_{1}(γ) on its domain C and

F_{2} : C × ℝ ↦ ℝ : (x, u) ↦ − log(u − F (x)) − ∑_{i=1}^{n} log x_{i}

satisfies the first condition of self-concordancy (2.2) with parameter κ_{2} = r_{2}(γ) on its domain epi F = {(x, u) | F (x) ≤ u}.

Note 2.5. A similar lemma is proved in [dJRT95], with parameters κ_{1} and κ_{2} both equal to 1 + γ. The second result is improved in [Jar96], with κ_{2} equal to max{1, γ}, as a special case of a more general compatibility theory developed in [NN94]. However, it is easy to see that our result is better. Indeed, our parameters are strictly lower in all cases for F_{1} and as soon as γ > 1 for r_{2}), with an asymptotical ratio of \sqrt{3} when γ tends to +∞.

Proof. We follow the lines of [dJRT95] and start with F_{1}: computing its second and third differentials gives

∇^{2}F_{1}(x)[h, h] = ∇^{2}F (x)[h, h] + ∑_{i=1}^{n} \frac{h_{i}^{2}}{x_{i}^{2}} and ∇^{3}F_{1}(x)[h, h, h] = ∇^{3}F (x)[h, h, h] − 2 ∑_{i=1}^{n} \frac{h_{i}^{3}}{x_{i}^{3}} .

Introducing two auxiliary variables a ≥ 0 and b ≥ 0 such that

a^{2} = ∇^{2}F [h, h] and b^{2} = ∑_{i=1}^{n} \frac{h_{i}^{2}}{x_{i}^{2}}

(convexity of F guarantees that a is real), we can rewrite inequality (2.15) as

∇^{3}F_{1}(x)[h, h, h] ≤ 3γa^{2}b .

Combining it with the fact that

|(∑_{i=1}^{n} \frac{h_{i}^{3}}{x_{i}^{3}} )^{\frac{1}{3}} | ≤ (∑_{i=1}^{n} \begin{matrix}|h_{i}| & 3 \\ |x_{i}| & 3 \end{matrix} )^{\frac{1}{3}} ≤ (∑_{i=1}^{n} \frac{h_{i}^{2}}{x_{i}^{2}} )^{\frac{1}{2}} = b ,  (2.16)

where the second inequality comes from the well-known relation ∥·∥_{3} ≤ ∥·∥_{2} applied to vector (\frac{h_{1}}{x_{1}} , …, \frac{h_{n}}{x_{n}}), we find that

\frac{∇^{3}F_{1}(x)[h, h, h]}{2(∇^{2}F_{1}(x)[h, h])^{\frac{3}{2}}} ≤ \frac{3γa^{2}b + 2b^{3}}{2(a^{2} + b^{2})^{\frac{3}{2}}} .

According to (2.2), finding the best parameter κ for F_{1} amounts to maximize this last quantity depending on a and b. Since a ≥ 0 and b ≥ 0 we can write a = r cos θ and b = r sin θ with r ≥ 0 and 0 ≤ θ ≤ \frac{π}{2} , which gives

\frac{3γa^{2}b + 2b^{3}}{2(a^{2} + b^{2})^{\frac{3}{2}}} = \frac{3γ}{2} cos^{2} θ sin θ + sin^{3} θ = h(θ) .

The derivative of h is

h′(θ) = \frac{3γ}{2} cos^{3} θ − 3γ sin^{2} θ cos θ + 3 cos θ sin^{2} θ = 3 cos θ (\frac{γ}{2} cos^{2} θ + (1 − γ) sin^{2} θ) .

When γ ≤ 1, this quantity is clearly always nonnegative, which implies that the maximum is attained for the largest value of θ, which gives h_{max} = h(\frac{π}{2} ) = 1 = r_{1}(γ). When γ < 1, we easily see that h has a maximum when \frac{γ}{2} cos^{2} θ + (1 − γ) sin^{2} = 0. This condition is easily seen to imply sin^{2} θ = \frac{γ}{3γ−2}, and h_{max} becomes

h_{max} = 3\frac{γ}{2} cos^{2} θ sin θ + sin^{3} θ = (3(γ − 1) + 1) sin^{3} θ
= (3γ − 2) (\frac{γ}{3γ − 2})^{\frac{3}{2}} = \frac{γ}{\sqrt{3 − 2/γ}} = r_{1}(γ) .

A similar but slightly more technical proof holds for F_{2}. Letting x̃ = (x, u), h̃ = (h, v) and G(x̃) = F (x) − u, we have that F_{2}(x̃) = − log(−G(x̃) − ∑_{i=1}^{n} log x_{i}. G is easily shown to be convex and negative on epi F, the domain of F_{2}. Since F and G only differ by a linear term, we also have that ∇^{2}F (x)[h, h] = ∇^{2}G(x̃)[h̃, h̃] and ∇^{3}F (x)[h, h, h] = ∇^{3}G(x̃)[h̃, h̃, h̃]. Looking now at the second differential of F_{2} we find

∇^{2}F_{2}(x̃)[h̃, h̃] = \frac{∇G(x̃)[h]^{2}}{G(x̃)^{2}} − \frac{∇^{2}G(x̃)[h̃, h̃]}{G(x̃)} + ∑_{i=1}^{n} \frac{h_{i}^{2}}{x_{i}^{2}} .

Let us define for convenience a ∈ ℝ_{+}, b ∈ ℝ_{+} and c ∈ ℝ with

a^{2} = −\frac{∇^{2}G(x̃)[h̃, h̃]}{G(x̃)}, b^{2} = ∑_{i=1}^{n} \frac{h_{i}^{2}}{x_{i}^{2}} and c = −\frac{∇G(x̃)[h]}{G(x̃)}

(convexity of G and the fact that it is negative on the domain of F_{2} guarantee that a is real), which implies ∇^{2}F_{2}(x)[h̃, h̃] = a^{2} + b^{2} + c^{2}. We can now evaluate the third differential

∇^{3}F_{2}(x̃)[h̃, h̃, h̃] = −\frac{∇^{3}G(x̃)[h̃, h̃, h̃]}{G(x̃)} + 3\frac{∇^{2}G(x̃)[h̃, h̃]∇G(x̃)[h̃]}{G(x̃)^{2}} − 2\frac{∇G(x̃)[h̃]^{3}}{G(x̃)^{3}} − 2 ∑_{i=1}^{n} \frac{h_{i}^{3}}{x_{i}^{3}}

= −\frac{∇^{3}G(x̃)[h̃, h̃, h̃]}{G(x̃)} + 3a^{2}c + 2c^{3} − 2 ∑_{i=1}^{n} \frac{h_{i}^{3}}{x_{i}^{3}}

≤ −\frac{∇^{3}G(x̃)[h̃, h̃, h̃]}{∇^{2}G(x̃)[h̃, h̃]} \frac{∇^{2}G(x̃)[h̃, h̃]}{G(x̃)} + 3a^{2}c + 2c^{3} + 2b^{3} using again (2.16)
= −\frac{∇^{3}F (x)[h, h, h]}{∇^{2}F (x)[h, h]} \frac{∇^{2}G(x̃)[h̃, h̃]}{G(x̃)} + 3a^{2}c + 2c^{3} + 2b^{3}
≤ 3γa^{2}b + 3a^{2}c + 2c^{3} + 2b^{3} using condition (2.15) .

According to (2.2), finding the best parameter κ for F_{2} amounts to maximize the following ratio

\frac{∇^{3}F_{2}(x̃)}{2(∇^{3}F_{2}(x̃))^{\frac{3}{2}}} ≤ \frac{3γa^{2}b + 3a^{2}c + 2c^{3} + 2b^{3}}{2(a^{2} + b^{2} + c^{2})^{\frac{3}{2}}} = \frac{\frac{3γ}{2} a^{2}b + \frac{3}{2}a^{2}c + c^{3} + b^{3}}{(a^{2} + b^{2} + c^{2})^{\frac{3}{2}}} .

Since this last quantity is homogeneous of degree 0 with respect to variables a, b and c, we can assume that a^{2} + b^{2} + c^{2} = 1, which gives

\frac{3γ}{2} a^{2}b + \frac{3}{2}a^{2}c + c^{3} + b^{3} = \frac{3}{2}a^{2}(γb + c) + c^{3} + b^{3} = \frac{3}{2}(1 − b^{2} − c^{2})(γb + c) + b^{3} + c^{3} .

Calling this last quantity m(b, c), we can now compute its partial derivatives with respect to b and c and find

\frac{∂m}{∂b} = −\frac{3}{2} ((3γ − 2)b^{2} + γc^{2} + 2bc − γ) and \frac{∂m}{∂c} = −\frac{3}{2} (b^{2} + c^{2} + 2bcγ − 1) .

We have now to equate those two quantities to zero and solve the resulting system. We can for example write \frac{∂m}{∂b} − γ \frac{∂m}{∂c} = 0, which gives (g − 1)b(b − c(γ + 1)) = 0, and explore the resulting three cases. The solutions we find are

(b, c) = (0, ±1) and (\frac{γ + 1}{\sqrt{3γ^{2} + 4γ + 2}}, \frac{1}{\sqrt{3γ^{2} + 4γ + 2}})

with an additional special case b + c = 1 when γ = 1. Plugging these values into m(b, c), one finds after some computations the following potential maximum values

±1 and \frac{γ^{2} + γ + 1}{\sqrt{3γ^{2} + 4γ + 2}} = \frac{γ + 1 + 1/γ}{\sqrt{3 + +4/γ + 2/γ^{2}}}

(and 1 in the special case γ = 1). One concludes that the maximum we seek is equal to r_{2}(γ), as announced.

While the lemma we have just proved is useful to tackle the first condition of selfconcordancy (2.2), it does not say anything about the second condition (2.3). The following Corollary about the second barrier F_{2} might prove useful in this respect.

Corollary 2.1. Let F satisfy the assumptions of Lemma 2.3. Then the second barrier

F_{2} : C × ℝ ↦ ℝ : (x, u) ↦ − log(u − F (x)) − ∑_{i=1}^{n} log x_{i}

is (r_{2}(γ), n + 1)-self-concordant.

Proof. Since G(x, u) = F (x) − u is convex, − log(u − F (x)) = − log(−G(x, u)) is known to satisfy the second self-concordancy condition (2.3) with ν = 1 by virtue of Lemma 2.1. Moreover, it is straightforward to check that each term − log x_{i} also satisfies that second condition with parameter ν = 1. Using the addition Theorem 2.7 and combining with the result of Lemma 2.3, we can conclude that F_{2} is (r_{2}(γ), n + 1)-self-concordant.

Note 2.6. We would like to point out that no similar result can hold for the first function F_{1}, since we know nothing about the status of the second self-concordancy condition (2.3) on its first term F (x). Indeed, taking the case of F : ℝ_{+} ↦ ℝ : x ↦ \frac{1}{x}, we can check that ∇^{2}F (x)[h, h] = 2\frac{h^{2}}{x^{3}} and ∇^{3}F (x)[h, h, h] = −6\frac{h^{3}}{x^{4}} , which implies that condition (2.15) holds with γ = 1 since

−6\frac{h^{3}}{x^{4}} ≤ 3 × 2\frac{h^{2}}{x^{3}} \frac{|h|}{x} ⇔ −h^{3} ≤ h^{2} |h|

is satisfied. On the other hand, the second self-concordancy condition (2.3) cannot hold for F_{1} : ℝ_{+} ↦ R : x ↦ \frac{1}{x} − log x, since

∇F (x)^{T} (∇^{2}F (x))^{−1}∇F (x) = \frac{F′_{1}(x)^{2}}{F′′_{1}(x)} = \frac{\frac{(x+1)^{2}}{x^{4}}}{\frac{(2+x)}{x^{3}}} = \frac{(x + 1)^{2}}{x(x + 2)}

does not admit an upper bound (it tends to +∞ when x → 0).

To conclude this section, we mention that since condition (2.15) is invariant with respect to positive scaling of F, the results from Lemma 2.3 hold for barriers F_{λ,1}(x) = λF (x) − ∑_{i=1}^{n} log x_{i} and F_{λ,2}(x, u) = − log(u − λF (x)) − ∑_{i=1}^{n} log x_{i} where λ is a positive constant.

2.4 Application to structured convex problems

In this section we rely on the work in [dJRT95], where several classes of structured convex optimization problems are shown to admit a self-concordant logarithmic barrier. However, Lemma 2.3 will allow us to improve the self-concordancy parameters and lower the resulting complexity values.

2.4.1 Extended entropy optimization

Let c ∈ ℝ^{n}, b ∈ ℝ^{m} and A ∈ ℝ^{m×n}. We consider the following problem

inf_{x∈ℝ^{n}} c^{T} x + ∑_{i=1}^{n} g_{i}(x_{i}) s.t. Ax = b and x ≥ 0  (EEO)

where scalar functions g_{i} : ℝ_{+} ↦ ℝ are required to satisfy

|g′′′_{i}(x)| ≤ κ_{i} \frac{g′′_{i}(x)}{x} ∀x ≥ 0  (2.17)

(which by the way implies their convexity). Classical entropy optimization results as a special case when g_{i}(x) = x log x (in that case, it is straightforward to see that condition (2.17) holds with κ_{i} = 1).

Let us use Lemma 2.3 with F_{i} : x_{i} ↦ g_{i}(x_{i}) and γ = \frac{κ_{i}}{3} . Indeed, checking condition (2.15) amounts to write

h^{3}g′′′_{i}(x) ≤ 3\frac{κ_{i}}{3} h^{2}g′′_{i}(x)\frac{|h|}{x} ⇔ \frac{h}{|h|}g′′′_{i}(x) ≤ κ_{i} \frac{g′′_{i}(x)}{x} ,

which is guaranteed by condition (2.17). Using the second barrier and Corollary 2.1, we find that

F_{i} : (x_{i}, u_{i}) ↦ − log(u_{i} − g_{i}(x_{i})) − log x_{i}

is (r_{2}(\frac{κ_{i}}{3} ), 2)-self-concordant^{2}. However, in order to use this barrier to solve problem (EEO), we need to reformulate it as

inf_{x∈ℝ^{n}} c^{T} x + ∑_{i=1}^{n} u_{i} s.t. Ax = b, g_{i}(x_{i}) ≤ u_{i} ∀1 ≤ i ≤ n and x ≥ 0 ,

which is clearly equivalent. We are now able to write the complete logarithmic barrier

F : (x, u) ↦ − ∑_{i=1}^{n} log(u_{i} − g_{i}(x_{i})) − ∑_{i=1}^{n} log x_{i} ,

which is (r_{2}(\frac{max\{κ_{i}\}}{3}), 2n)-self-concordant by virtue of Theorem 2.7. In light of Note 2.4, we can even do better with a different scaling of each term, to get

F̃ : (x, u) ↦ − ∑_{i=1}^{n} r_{2}(\frac{κ_{i}}{3} )^{2} log(u_{i} − g_{i}(x_{i})) − ∑_{i=1}^{n} r_{2}(\frac{κ_{i}}{3} )^{2} log x_{i}

which is then (1, \sqrt{2 ∑_{i=1}^{n} r_{2}(\frac{κ_{i}}{3} )^{2}})-self-concordant. In the case of classical entropy optimiza-

tion, these parameters become (1, \sqrt{2n}) , since r_{2}(\frac{1}{3}) = 1.

2.4.2 Dual geometric optimization

Let {I_{k}}_{k=1...r} be a partition of {1, 2, …, n}, c ∈ ℝ^{n}, b ∈ ℝ^{m} and A ∈ ℝ^{m×n}. The dual geometric optimization problem is (see Chapter 5 for a complete description)

inf_{x∈ℝ^{n}} c^{T} x + ∑_{k=1}^{r} [∑_{i∈I_{k}} x_{i} log(\frac{x_{i}}{∑_{i∈I_{k}} x_{i}})]
s.t. Ax = b and x ≥ 0  (GD)

It is shown in [dJRT95] that condition (2.15) holds for

F_{k} : (x_{i})_{i∈I_{k}} ↦ ∑_{i∈I_{k}} x_{i} log(\frac{x_{i}}{∑_{i∈I_{k}} x_{i}})

with γ = 1, so that the corresponding second barrier in Lemma 2.15 is (1, |I_{k}| + 1)-self-concordant. Using the same trick as for problem (EEO), we introduce additional variables u_{k} to find that the following barrier

F : (x, u) ↦ ∑_{k=1}^{r} − log(u_{k} − ∑_{i∈I_{k}} x_{i} log(\frac{x_{i}}{∑_{i∈I_{k}} x_{i}})) − ∑_{i=1}^{n} log x_{i}

is a (1, n + r)-self-concordant barrier for a suitable reformulation of problem (GD).

2This corrects the statement in [dJRT95] where it is mentioned that g_{i}(x_{i}) − log x_{i}, i.e. the first barrier in Lemma 2.3, is self-concordant. As it is made clear in Note 2.6, this cannot be true in general

2.4.3 l_{p}-norm optimization

Let {I_{k}}_{k=1...r} be a partition of {1, 2, …, n}, b ∈ ℝ^{m}, a_{i} ∈ ℝ^{m}, f_{k} ∈ ℝ^{m}, c ∈ ℝ^{n}, d ∈ ℝ^{r} and p ∈ ℝ^{n} such that p_{i} ≥ 1. The primal l_{p}-norm optimization problem is (see Chapter 4 for a complete description)

sup_{y∈ℝ^{m}} b^{T} y s.t. f_{i}(y) ≤ 0 for all k = 1, …, r ,  (Pl_{p})

where functions f_{i} : ℝ^{m} ↦ ℝ are defined according to

f_{i} : y ↦ ∑_{i∈I_{k}} \frac{1}{p_{i}} |a_{i}^{T} y − c_{i}|^{p_{i}} + f_{k}^{T} y − d_{k} .

This problem can be reformulated as

sup_{y∈ℝ^{m}} b^{T} y s.t. \{ \begin{matrix}|a_{i}^{T} y − c_{i}| & ≤ & s_{i} & ∀i = 1, …, n \\ s_{i} & ≤ & t_{i}^{1/p_{i}} & ∀i = 1, …, n \\ ∑i∈I_{k} \frac{t_{i}}{p_{i}} & ≤ & d_{k} − f_{k}^{T} y & ∀k = 1, …, r \end{matrix}

where each of the m constraints involving an absolute value is indeed equivalent to a pair of linear constraints a_{i}^{T} y − c_{i} ≤ s_{i} and c_{i} − a_{i}^{T} y ≤ s_{i}. Once again, a self-concordant function can be found for the difficult part of the constraints, i.e. the nonlinear inequality s_{i} ≤ t_{i}^{1/p_{i}}. Indeed, it is straightforward to check that f_{i} : t_{i} ↦ −t_{i}^{1/p_{i}} satisfies condition (2.15) with γ = \frac{2p_{i}−1}{3p_{i}} < 1, which implies in the same fashion as above that

− log(t_{i}^{1/p_{i}} − s_{i}) − log t_{i}

is (1, 2)-self-concordant. Combining with the logarithmic barrier for the linear constraints, we have that

− ∑_{i=1}^{m} log(s_{i} − a_{i}^{T} y + c_{i}) − ∑_{i=1}^{m} log(s_{i} + a_{i}^{T} y − c_{i}) − ∑_{i=1}^{m} log(t_{i}^{1/p_{i}} − s_{i}) …

… − ∑_{i=1}^{m} log t_{i} − ∑_{k=1}^{r} log(d_{k} − f_{k}^{T} y − ∑_{i∈I_{k}} \frac{t_{i}}{p_{i}} )

is (1, 4m + r)-self-concordant for our reformulation of problem (Pl_{p}) (since each linear constraint is (1, 1)-self-concordant).

Let us mention that another reformulation is presented in [dJRT95], where Lemma 2.3 is applicable to the nonlinear constraint with parameter γ = \frac{|p_{i}−2|}{3}, with the disadvantage of having a parameter that depends on p_{i} (although r_{2}(γ) will stay at its lowest value as long as p_{i} ≤ 5).

We conclude this section by mentioning that very similar results hold for the dual l_{p}-norm optimization problem, and we refer the reader to [dJRT95] for the details^{3}.

3However, we would like to point out that the nonlinear function involved in these developments is wrongly stated to satisfy condition (2.15) with γ = \frac{\sqrt{2}(q_{i}+1)}{3q_{i}}, while a correct value is ^{\frac{\sqrt{5q_{i}^{2}−2q_{i}+2}}{3q_{i}}}.

2.5 Concluding remarks

We gave in this chapter an overview of the theory of self-concordant functions. We would like to point out that this very powerful framework relies on two different conditions (2.2) and (2.3) and the two corresponding parameters κ and ν, each with its own purpose (see the discussion in Note 2.3). However, the important quantity is the resulting complexity value κ\sqrt{ν}, which is of the same order as the number of iterations that is needed to reduce the barrier parameter by a constant factor by the short-step interior-point algorithm.

It is possible to scale self-concordant barriers such that one of the parameters is arbitrarily fixed without any real loss of generality. We have shown that this is best done fixing parameter κ, considering the way the complexity value is affected when adding several self-concordant barriers. However, it is in our opinion better to keep two parameters all the time, in order to simplify the presentation (for example, Lemma 2.3 intrinsically deals with the κ parameter and would need a rather awkward reformulation to be written for parameter ν with κ fixed to 1).

Several important results help us prove self-concordancy of barrier functions: Lemmas 2.1 and 2.2 deal with the second self-concordancy condition (2.3), while our improved Lemma 2.3 pertains to the first self-concordancy condition (2.2). They are indeed responsible for most of the analysis carried out in Section 2.4, which is dedicated to several classes of structured convex optimization problems. Namely, it is proved that nearly all the nonlinear (i.e. corresponding to the nonlinear constraints) terms in the associated logarithmic barriers are self-concordant with κ = 1 (the exception being extended entropy optimization, which encompasses a very broad class of problems). We would also like to mention that since all the barriers that are presented are polynomially computable, as well as their gradient and Hessian, the short-step method applied to any of these problems would need to perform a polynomial number of arithmetic operations to provide a solution with a given accuracy.

To conclude, we would like to speculate on the possibility of replacing the two selfconcordancy conditions by a single inequality. Indeed, since the complexity value κ\sqrt{ν} is the only quantity that really matters in the final complexity result, one could imagine to consider the following inequality

\frac{F′′′_{x,h}(0)F′_{x,h}(0)}{F′′_{x,h}(0)^{2}} ≤ 2C for all x ∈ int C and h ∈ ℝ^{n} ,  (2.18)

which is satisfied with C = κ\sqrt{ν} for (κ, ν)-self-concordant barriers (to see that, simply multiply condition (2.5b) by the square root of condition (2.6b)). We point out the following two intriguing facts and leave their investigation for further research:

⋄ Condition (2.18) appears to be central in the recent theory of self-regular functions [PRT00], an attempt at generalizing self-concordant functions.

⋄ Following the same principles as for (2.5d) and (2.6d), condition (2.18) can be reformulated as

(−\frac{F′_{x,h}(t)}{F′′_{x,h}(t)})′ ≤ 2C − 1 ,

where the quantity on the left-hand side is the derivative of the Newton step applied to the restriction F_{x,h}.

Part II

Conic Duality

Conic optimization

In this section, we describe conic optimization and the associated duality theory. Conic optimization deals with a class of problems that is essentially equivalent to the class of convex problems, i.e. minimization of a convex function over a convex set. However, formulating a convex problem in a conic way has the advantage of providing a very symmetric form for the dual problem, which often gives a new insight about its structure, especially dealing with duality.

3.1 Conic problems

The results we present in this Chapter are well-known and we will skip most of the proofs. They can be found for example in the Ph.D. thesis of Sturm [Stu97, Stu99a] with similar notations, more classical references presenting equivalent results are [SW70] and [ET76, Chapter III, Section 5]).

The basic ingredient of conic optimization is a convex cone.

Definition 3.1. A set C is a cone if and only if it is closed under nonnegative scalar multiplication, i.e.

x ∈ C ⇒ λx ∈ C for all λ ∈ ℝ_{+} .

Recall that a set is convex if and only if it contains the whole segment joining any two of its points. Establishing convexity is easier for cones than for general sets, because of the following elementary theorem [Roc70a, Theorem 2.6]:

Theorem 3.1. A cone C is convex if and only if it is closed under addition, i.e.

x ∈ C and y ∈ C ⇒ x + y ∈ C .

In order to avoid some technical nuisances, the convex cones we are going to consider will be required to be closed, pointed and solid, according to the following definitions. A cone is said to be pointed if it doesn’t contain any straight line passing through the origin, which can be expressed as

Definition 3.2. A cone C is pointed if and only if C ∩ −C = {0}, where −C stands for the set {x | −x ∈ C}

Furthermore, a cone is said to be solid if it has a nonempty interior, i.e. it is fulldimensional.

Definition 3.3. A cone C is solid if and only if int C ≠ ∅ (where int S denotes the interior of set S).

For example, the positive orthant is a pointed and solid convex cone. A linear subspace is a convex cone that is neither pointed, nor solid (except ℝ^{n} itself).

We are now in position to define a conic optimization problem: let C ⊆ ℝ^{n} a pointed, solid, closed convex cone. The (primal) conic optimization problem is

inf_{x∈ℝ^{n}} c^{T} x s.t. Ax = b and x ∈ C ,  (CP)

where x ∈ ℝ^{n} is the column vector we are optimizing and the problem data is given by cone C, a m × n matrix A and two column vectors b and c belonging respectively to ℝ^{m} and ℝ^{n}. This problem can be viewed as the minimization of a linear function over the intersection of a convex cone and an affine subspace. As an illustration, let us mention that a linear optimization problem in the standard form (1.2) is formulated by choosing cone C to be the positive orthant ℝ_{+}^{n}.

At this stage, we would like to emphasize the fact that although our cone C is closed, it may happen that the infimum in (CP) is not attained (some examples of this situation will be given in Subsection 3.3).

It is well-known that the class of conic problems is equivalent to the class of convex problems, see e.g. [NN94]. However, the usual Lagrangean dual of a conic problem can be expressed very nicely in a conic form, using the notion of dual cone.

Definition 3.4. The dual of a cone C ⊆ ℝ^{n} is defined by

C^{∗} = {x^{∗} ∈ ℝ^{n} | x^{T} x^{∗} ≥ 0 for all x ∈ C} .

For example, the dual of ℝ_{+}^{n} is ℝ_{+}^{n} itself. We say it is self-dual. Another example is the dual of the linear subspace L, which is L^{∗} = L^{⊥}, the linear subspace orthogonal to L (note that in that case the inequality of Definition 3.4 is always satisfied with equality).

The following theorem stipulates that the dual of a closed convex cone is always a closed convex cone [Roc70a, Theorem 14.1].

Theorem 3.2. If C is a closed convex cone, its dual C^{∗} is another closed convex cone. Moreover, the dual (C^{∗})^{∗} of C^{∗} is equal to C.

Closedness is essential for (C^{∗})^{∗} = C to hold (without the closedness assumption on C, we only have (C^{∗})^{∗} = cl C where cl S denotes the closure of set S [Roc70a, Theorem 14.1]). The additional notions of solidness and pointedness also behave well when taking the dual of a convex cone: indeed, these two properties are dual to each other [Stu97, Corollary 2.1], which allows us to state the following theorem:

Theorem 3.3. If C is a solid, pointed, closed convex cone, its dual C^{∗} is another solid, pointed, closed convex cone.

The dual of our primal conic problem (CP) is defined by

sup_{y∈ℝ^{m},s∈ℝ^{n}} b^{T} y s.t. A^{T} y + s = c and s ∈ C^{∗} ,  (CD)

where y ∈ ℝ^{m} and s ∈ ℝ^{n} are the column vectors we are optimizing, the other quantities A, b and c being the same as in (CP). It is immediate to notice that this dual problem has the same kind of structure as the primal problem, i.e. it also involves optimizing a linear function over the intersection of a convex cone and an affine subspace. The only differences are the direction of the optimization (maximization instead of minimization) and the way the affine subspace is described (it is a translation of the range space of A^{T}, while primal involved a translation of the null space of A). It is also easy to show that the dual of this dual problem is equivalent to the primal problem, using the fact that (C^{∗})^{∗} = C.

One of the reasons the conic formulation (CP) is interesting is the fact that we may view the constraint x ∈ C as a generalization of the traditional nonnegativity constraint x ≥ 0 of linear optimization. Indeed, let us define the relation ⪰ on ℝ^{n} × ℝ^{n} according x ⪰ y ⇔ x − y ∈ C. This relation is reflexive, since x ⪰ x ⇔ 0 ∈ C is always true. It is also transitive, since we have

x ⪰ y and y ⪰ z ⇔ x − y ∈ C and y − z ∈ C ⇒ (x − y) + (y − z) = x − z ∈ C ⇔ x ⪰ z

(where we used the fact that a convex cone is closed under addition, see Theorem 3.1). Finally, using the fact that C is pointed, we can write

x ⪰ y and y ⪰ x ⇔ x − y ∈ C and − (x − y) ∈ C ⇒ x − y = 0 ⇒ x = y ,

which shows that relation ⪰ is antisymmetric and is thus a partial order on ℝ^{n} ×ℝ^{n}. Defining ⪰^{∗} to be the relation induced by the dual cone C^{∗}, we can rewrite our primal-dual pair (CP)– (CD) as

inf_{x∈ℝ^{n}} c^{T} x s.t. Ax = b and x ⪰ 0
sup_{y∈ℝ^{m}} b^{T} y s.t. c ⪰^{∗} A^{T} y ,

which looks very much like a generalization of the primal-dual pair of linear optimization problems (LP)–(LD’).

For example, one of the most versatile cones used in convex optimization is the positive semidefinite cone 𝕊_{+}^{n}.

Definition 3.5. The positive semidefinite cone 𝕊_{+}^{n} is a subset of 𝕊^{n}, the set of symmetric n × n matrices. It consists of all positive semidefinite matrices, i.e.

M ∈ 𝕊_{+}^{n} ⇔ z^{T} Mz ≥ 0 ∀z ∈ ℝ^{n} ⇔ λ(M) ≥ 0

where λ(M) denotes the vector of eigenvalues of M.

It is straightforward to check that 𝕊_{+}^{n} is a closed, solid, pointed convex cone. A conic optimization problem of the form (CP) or (CD) that uses a cone of the type 𝕊_{+}^{n} is called a semidefinite problem^{1}. This cone provides us with the ability to model many more types of constraints than a linear problem (see e.g. [VB96] or Appendix A for an application to classification).

3.2 Duality theory

The two conic problems of this primal-dual pair are strongly related to each other, as demonstrated by the duality theorems stated in this section. Conic optimization enjoys the same kind of rich duality theory as linear optimization, albeit with some complications regarding the strong duality property.

Theorem 3.4 (Weak duality). Let x a feasible (i.e. satisfying the constraints) solution for (CP), and (y, s) a feasible solution for (CD). We have

b^{T} y ≤ c^{T} x ,

equality occurring if and only if the following orthogonality condition is satisfied:

x^{T} s = 0 .

This theorem shows that any primal (resp. dual) feasible solution provides an upper (resp. lower) bound for the dual (resp. primal) problem. Its proof is quite easy to obtain: elementary manipulations give

c^{T} x − b^{T} y = x^{T} c − (Ax)^{T} y = x^{T} (A^{T} y + s) − x^{T} A^{T} y = x^{T} s ,

this last inner product being always nonnegative because of x ∈ C, s ∈ C^{∗} and Definition 3.4 of the dual cone C^{∗}. The nonnegative quantity x^{T} s = c^{T} x − b^{T} y is called the duality gap.

Obviously, a pair (x, y) with a zero duality gap must be optimal. It is well known that the converse is true in the case of linear optimization, i.e. that all primal-dual pairs of optimal solutions for a linear optimization problem have a zero duality gap (see Section 1.2.1), but this is not in general the case for conic optimization.

1The fact that our feasible points are in this case matrices instead of vectors calls for some explanation.

Since our convex cones are supposed to belong to a real vector space, we have to consider that 𝕊^{n}, the space of symmetric matrices, is isomorphous to ℝ^{n(n+1)/2}. In that setting, an expression such as the objective function c^{T} x, where c and x belong to ℝ^{n(n+1)/2}, is to be understood as the inner product of the corresponding symmetric matrices C and X in the space 𝕊^{n}, which is defined by ⟨C, X⟩ = trace CX. Moreover, A can be seen in this case as an application (more precisely a tensor) that maps 𝕊^{n} to ℝ^{m}, while A^{T} is the adjoint of A which maps ℝ^{m} to 𝕊^{n}.

Denoting by p^{∗} and d^{∗} the optimum objective values of problems (CP) and (CD), the previous theorem implies that p^{∗} − d^{∗} ≥ 0, a nonnegative quantity which will be called the optimal duality gap. Under certain circumstances, it can be proved to be equal to zero, which shows that the optimum values of problems (CP) and (CD) are equal. Before describing the conditions guaranteeing such a situation, called strong duality, we need to introduce the notion of strictly feasible point.

Definition 3.6. A point x (resp. (y, s)) is said to be strictly feasible for the primal (resp. dual) problem if and only if it is feasible and belongs to the interior of the cone C (resp. C^{∗}), i.e.

Ax = b and x ∈ int C (resp. A^{T} y + s = c and s ∈ int C^{∗}) .

Strictly feasible points, sometimes called Slater points, are also said to satisfy the interiorpoint or Slater condition. Moreover, we will say that the primal (resp. dual) problem is unbounded if p^{∗} = −∞ (resp. d^{∗} = +∞), that it is infeasible if there is no feasible solution, i.e. when p^{∗} = +∞ (resp. d^{∗} = −∞), and that it is solvable or attained if the optimum objective value p^{∗} (resp. d^{∗}) is achieved by at least one feasible primal (resp. dual) solution.

Theorem 3.5 (Strong duality). If the dual problem (CD) admits a strictly feasible solution, we have either

⋄ an infeasible primal problem (CP) if the dual problem (CD) is unbounded, i.e. p^{∗} = d^{∗} = +∞

⋄ a feasible primal problem (CP) if the dual problem (CD) is bounded. Moreover, in this case, the primal optimum is finite and attained with a zero duality gap, i.e. there is at least an optimal feasible solution x^{∗} such that c^{T} x^{∗} = p^{∗} = d^{∗}.

The first case in this theorem (see e.g. [Stu97, Theorem 2.7] for a proof) is a simple consequence of Theorem 3.4, which is also valid in the absence of a Slater point for the dual, as opposed to the second case which relies on the existence of such a point. It is also worth to mention that boundedness of the dual problem (CD), defining the second case, is implied by the existence of a feasible primal solution, because of the weak duality theorem (however, the converse implication is not true in general, since a bounded dual problem can admit an infeasible primal problem ; an example of this situation is provided in Subsection 5.3.4).

This theorem is important, because it provides us with way to identify when both the primal and the dual problems have the same optimal value, and when this optimal value is attained by one of the problems. Obviously, this result can be dualized, meaning that the existence of a strictly feasible primal solution implies a zero duality gap and dual attainment. The combination of these two theorems leads to the following well-known corollary:

Corollary 3.1. If both the primal and the dual problems admit a strictly feasible point, we have a zero duality gap and attainment for both problems, i.e. the same finite optimum objective value is attained for both problems.

When the dual problem has no strictly feasible point, nothing can be said about the duality gap (which can happen to be strictly positive) and about attainment of the primal optimum objective value. However, even in this situation, we can prove an alternate version of the strong duality theorem involving the notion of primal problem subvalue. The idea behind this notion is to allow a small constraint violation in the infimum defining the primal problem (CP).

Definition 3.7. The subvalue of primal problem (CP) is given by

p^{−} = lim_{ϵ→0^{+}} [ inf_{x} c^{T} x s.t. ∥Ax − b∥ < ϵ and x ∈ C ]

(a similar definition is holding for the dual subvalue d^{−}).

It is readily seen that this limit always exists (possibly being +∞), because the feasible region of the infimum shrinks as ϵ tends to zero, which implies that its optimum value is a nonincreasing function of ϵ. Moreover, the inequality p^{−} ≤ p^{∗} holds, because all the feasible regions of the infima defining p^{−} as ϵ tends to zero are larger than the actual feasible region of problem (CP).

The case p^{−} = +∞, which implies that primal problem (CP) is infeasible (since we have then p^{∗} ≥ p^{−} = +∞), is called primal strong infeasibility, and essentially means that the affine subspace defined by the linear constraints Ax = b is strongly separated from cone C. We are now in position to state the following alternate strong duality theorem:

Theorem 3.6 (Strong duality, alternate version). We have either

⋄ p^{−} = +∞ and d^{∗} = −∞ when primal problem (CP) is strongly infeasible and dual problem (CD) is infeasible.

⋄ p^{−} = d^{∗} in all other cases.

This theorem (see e.g. [Stu97, Theorem 2.6] for a proof) states that there is no duality gap between p^{−} and d^{∗}, except in the rather exceptional case of primal strong infeasibility and dual infeasibility. Note that the second case covers situations where the primal problem is infeasible but not strongly infeasible (i.e. p^{−} < p^{∗} = +∞).

To conclude this section, we would like to mention the fact that all the properties and theorems described in this section can be easily extended to the case of several conic constraints involving disjoint sets of variables. Note 3.1. Namely, having to satisfy the constraints x^{i} ∈ C^{i} for all i ∈ {1, 2, …, k}, where C^{i} ⊆ ℝ^{n_{i}}, we will simply consider the Cartesian product of these cones C = C^{1}×C^{2}×⋯×C^{k} ⊆ ℝ^{∑_{i=1}^{k} n_{i}} and express all these constraints simultaneously as x ∈ C with x = (x^{1}, x^{2}, …, x^{k}). The dual cone of C will be given by

C^{∗} = (C^{1})^{∗} × (C^{2})^{∗} × ⋯ × (C^{k})^{∗} ⊆ ℝ^{∑_{i=1}^{k} n_{i}} ,

as implied by the following theorem:

Theorem 3.7. Let C^{1} and C^{2} two closed convex cones, and C = C^{1} × C^{2} their Cartesian product. Cone C is also a closed convex cone, and its dual C^{∗} is given by

C^{∗} = (C^{1})^{∗} × (C^{2})^{∗} .

3.3 Classification of conic optimization problems

In this last section, we describe all the possible types of conic programs with respect to feasibility, attainability of the optimum and optimal duality gap, and provide corresponding examples.

Given our standard primal conic program (CP), we define

F_{+} = {x ∈ ℝ^{n} | Ax = b and x ∈ C}

to be its feasible set and δ = dist(C, L) the minimum distance between cone C and the affine subspace L = {x | Ax = b} defined by the linear constraints. We also call F_{++} the set of strictly feasible solutions of (CP), i.e.

F_{++} = {x ∈ ℝ^{n} | Ax = b and x ∈ int C} .

3.3.1 Feasibility

First of all, the distinction between feasible and infeasible conic problems is not as clear-cut as for linear optimization. We have the following cases^{2}

⋄ A conic program is infeasible. This means the feasible set F_{+} = ∅, and that p^{∗} = +∞. But we have to distinguish two subcases

– δ = 0, which means an infinitesimal perturbation of the problem data may transform the program into a feasible one. We call the program weakly infeasible(‡). This corresponds to the case of a finite subvalue, i.e. p^{−} < p^{∗} = +∞.

– δ > 0, which corresponds to the usual infeasibility as for linear optimization. We call the program strongly infeasible, which corresponds to an infinite subvalue p^{−} = p^{∗} = +∞.

⋄ A conic program is feasible, which means F_{+} ≠ ∅ and p^{∗} < +∞ (and thus δ = 0). We also distinguish two subcases

– F_{++} = ∅, which implies that all feasible points belong to the boundary of the feasible set F_{+} (this corresponds indeed to the case where the affine subspace L is tangent to the cone C). This also means that an infinitesimal perturbation of the problem data can make the program infeasible. We call the program weakly feasible.

– F_{++} ≠ ∅. We call the program strongly feasible. This means there exists at least one feasible solution belonging to the interior of C, which is the main hypothesis of the strong duality Theorem 3.5.

It is possible to characterize these situations by looking at the existence of certain types of directions in the dual problem (level direction, improving direction, improving direction sequence, see [Stu97]). Let us now illustrate these four situations with an example.

2In the following, we’ll mark with a (‡) the cases which never happen in the case of linear optimization.

Example 3.1. Let us choose

C = 𝕊_{+}^{2} and x = (\begin{matrix}x_{1} & x_{3} \\ x_{3} & x_{2} \end{matrix}) .

We have that x ∈ C ⇔ x_{1} ≥ 0, x_{2} ≥ 0 and x_{1}x_{2} ≥ x_{3}^{2}.

If we add the linear constraint x_{3} = 1, the feasible set becomes the epigraph of the positive branch of the hyperbola x_{1}x_{2} = 1, i.e. F_{+} = {(x_{1}, x_{2}) | x_{1} ≥ 0 and x_{1}x_{2} ≥ 1} as depicted on Figure 3.1.

Figure 3.1: Epigraph of the positive branch of the hyperbola x_{1}x_{2} = 1

This problem is strongly feasible.

⋄ If we add another linear constraint x_{1} = −1, we get a strongly infeasible problem (since x_{1} must be positive).

⋄ If we add x_{1} = 0, we get a weakly infeasible problem (since the distance between the axis x_{1} = 0 and the hyperbola is zero but x_{1} still must be positive).

⋄ Finally, adding x_{1} +x_{2} = 2 leads to a weakly feasible problem (because the only feasible point, x_{1} = x_{2} = x_{3} = 1, does not belong to the interior of C).

3.3.2 Attainability

Let us denote by F^{∗} the set of optimal solutions, i.e. feasible solutions with an objective equal to p^{∗}

F^{∗} = F_{+} ∩ {x ∈ ℝ^{n} | c^{T} x = p^{∗}}

We have the following distinction regarding attainability of the optimum

⋄ A conic program is solvable if F^{∗} ≠ ∅.

⋄ A conic program is unsolvable if F^{∗} = ∅, but we have two subcases

– If p^{∗} = −∞, the program is unbounded (this is the only possibility in the case of linear optimization).

– If p^{∗} is finite, we have a feasible unsolvable bounded program (‡). This situation happens when the infimum defining p^{∗} is not attained, i.e. there exists feasible solution with objective value arbitrarily close to p^{∗} but no optimal solution.

Let us examine a little further the second situation. In this case, we have a sequence of feasible solutions whose objective value tends to p^{∗}, but no optimal solution. This implies that at least one of the variables in this sequence of feasible solutions tends to infinity. Indeed, if it was not the case, that sequence would be bounded, and since the feasible set F is closed (it is the intersection of a closed cone and a affine subspace, which is also closed), its limit would also belong to the feasible set, hence would be a feasible solution with objective value p^{∗}, i.e. an optimal solution, which is a contradiction.

Example 3.2. Let us consider the same strongly feasible problem as in Example 3.1 (epigraph of an hyperbola).

⋄ If we choose a linear objective equal to x_{1} + x_{2},, F^{∗} is reduced to the unique point (x_{1}, x_{2}, x_{3}) = (1, 1, 1), and the problem is solvable (p^{∗} = 2).

⋄ If we choose another objective equal to −x_{1} − x_{2}, F^{∗} = ∅ because p^{∗} = −∞, and the problem is unbounded.

⋄ Finally, choosing x_{1} as objective function leads to an unsolvable bounded problem: p^{∗} is easily seen to be equal to zero but F^{∗} = ∅ because there is no feasible solution with x_{1} = 0 since the product x_{1}x_{2} has to be greater than 1.

3.3.3 Optimal duality gap

Finally, we state the various possibilities about the optimal duality gap, which is equal to p^{∗} − d^{∗}:

⋄ The optimal duality gap is strictly positive (‡)

⋄ The optimal duality gap is zero but there is no optimal solution pair. In this case, there exists pairs (x, y) with an arbitrarily small duality gap (which means that the optimum is not attained for at least one of the two programs (LP) and (LD)) (‡)

⋄ An optimal solution pair (x, y) has a zero duality gap, as for linear optimization

Of course, the first two cases can be avoided if we require our problem to satisfy the Slater condition. We can alternatively work with the subvalue p^{−}, for which there is no duality gap except when both problems are infeasible.

Example 3.3. The first problem described in Example 3.2 has its optimal value equal to p^{∗} = 2. Its data can be described as

c = (\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix}) , A : 𝕊^{2} ↦ ℝ : (\begin{matrix}x_{1} & x_{3} \\ x_{3} & x_{2} \end{matrix}) ↦ x_{3} and b = 1 .

Using the fact that the adjoint of A can be written as^{3}

A^{T} : ℝ ↦ 𝕊^{2} : y_{1} ↦ (\begin{matrix}0 & y_{1}/2 \\ y_{1}/2 & 0 \end{matrix})

and the dual formulation (CD), we can state the dual as

sup y_{1} s.t. (\begin{matrix}0 & y_{1}/2 \\ y_{1}/2 & 0 \end{matrix}) + (\begin{matrix}s_{1} & s_{3} \\ s_{3} & s_{2} \end{matrix}) = (\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix}) and (\begin{matrix}s_{1} & s_{3} \\ s_{3} & s_{2} \end{matrix}) ∈ 𝕊_{+}^{2}

or equivalently, after eliminating the s variables,

sup y_{1} s.t. (\begin{matrix}1 & −y_{1}/2 \\ −y_{1}/2 & 1 \end{matrix}) ∈ 𝕊_{+}^{2} .

The optimal value d^{∗} of this problem is equal to 2, because the semidefinite constraint is equivalent to y_{1}^{2} ≤ 4), and the optimal duality gap p^{∗} − d^{∗} is zero as expected.

Changing the primal objective to c = (\begin{matrix}1 & 0 \\ 0 & 0 \end{matrix}), we get an unsolvable bounded problem

inf x_{1} s.t. x_{3} = 1 and x_{1}x_{2} ≥ 1

whose optimal value is p^{∗} = 0 but is not attained. The dual becomes

sup y_{1} s.t. (\begin{matrix}1 & −y_{1}/2 \\ −y_{1}/2 & 0 \end{matrix}) ∈ 𝕊_{+}^{2}

which admits only one feasible solution, namely y_{1} = 0, and has thus an optimal value d^{∗} = 0. In this case, the optimal duality gap is zero but is not attained (because the primal problem is unsolvable).

Finally, we give here an example where the optimal duality gap is nonzero. Choosing a nonnegative parameter λ and

C = 𝕊_{+}^{3}, c = (\begin{matrix}0 & −1 & 0 \\ −1 & 0 & 0 \\ 0 & 0 & λ \end{matrix}) , A : 𝕊^{3} ↦ ℝ^{2} : (\begin{matrix}x_{1} & x_{4} & x_{5} \\ x_{4} & x_{2} & x_{6} \\ x_{5} & x_{6} & x_{3} \end{matrix}) ↦ (\begin{matrix}x_{3} + x_{4} \\ x_{2} \end{matrix}) and b = (\begin{matrix}1 \\ 0 \end{matrix}) ,

we have for the primal

inf λx_{3} − 2x_{4} s.t. x_{3} + x_{4} = 1, x_{2} = 0 and (\begin{matrix}x_{1} & x_{4} & x_{5} \\ x_{4} & x_{2} & x_{6} \\ x_{5} & x_{6} & x_{3} \end{matrix}) ∈ 𝕊_{+}^{3} .

3To check this, simply write ⟨Ax, y⟩ = ⟨x, A^{T} y⟩, where the first inner product is the usual dot product on ℝ^{n} but the second inner product is the trace inner product on 𝕊^{n}.

The fact that x_{2} = 0 implies x_{4} = x_{6} = 0, which in turn implies x_{3} = 1. We have thus that all solutions have the form

(\begin{matrix}x_{1} & 0 & x_{5} \\ 0 & 0 & 0 \\ x_{5} & 0 & 1 \end{matrix})

which is feasible as soon as x_{1} ≥ x_{5}^{2}. All these feasible solutions have an objective value equal λ, and hence are all optimal: we have p^{∗} = λ. Using the fact that the adjoint of A is

A^{T} : ℝ^{2} ↦ 𝕊^{3} : (\begin{matrix}y_{1} \\ y_{2} \end{matrix}) ↦ (\begin{matrix}0 & y_{1}/2 & 0 \\ y_{1}/2 & y_{2} & 0 \\ 0 & 0 & y_{1} \end{matrix})

we can write the dual (after eliminating the s variables with the linear equality constraints) as

sup y_{1} s.t. (\begin{matrix}0 & −1 − y_{1}/2 & 0 \\ −1 − y_{1}/2 & −y_{2} & 0 \\ 0 & 0 & λ − y_{1} \end{matrix}) ∈ 𝕊_{+}^{3}

The above matrix can only be positive semidefinite if y_{1} = −2. In that case, any nonnegative value for y_{2} will lead to a feasible solution with an objective equal to −2, i.e. all these solutions are optimal and d^{∗} = −2. The optimal duality gap is equal to p^{∗} −d^{∗} = λ+2, which is strictly positive fear all values of λ. Note that in this case, as expected from the theory, none of the two problems satisfies the Slater condition since every feasible primal or dual solution has at least a zero on its diagonal, which implies a zero eigenvalue and hence that it does not belong to the interior of 𝕊_{+}^{3}.

l_{p}-norm optimization

In this chapter, we formulate the l_{p}-norm optimization problem as a conic optimization problem, derive its standard duality properties and show it can be solved in polynomial time.

We first define an ad hoc closed convex cone L^{p}, study its properties and derive its dual. This allows us to express the standard l_{p}-norm optimization primal problem as a conic problem involving L^{p}. Using the theory of conic duality described in Chapter 3 and our knowledge about L^{p}, we proceed to derive the dual of this problem and prove the well-known regularity properties of this primaldual pair, i.e. zero duality gap and primal attainment. Finally, we prove that the class of l_{p}-norm optimization problems can be solved up to a given accuracy in polynomial time, using the framework of interior-point algorithms and self-concordant barriers.

4.1 Introduction

l_{p}-norm optimization problems form an important class of convex problems, which includes as special cases linear optimization, quadratically constrained convex quadratic optimization and l_{p}-norm approximation problems.

74  4. l_{p}-norm optimization

A few interesting duality results are known for l_{p}-norm optimization. Namely, a pair of feasible primal-dual l_{p}-norm optimization problems satisfies the weak duality property, which is a mere consequence of convexity, but can also be shown to satisfy two additional properties that cannot be guaranteed in the general convex case: the optimum duality gap is equal to zero and at least one feasible solution attains the optimum primal objective. These results were first presented by Peterson and Ecker [PE70a, PE67, PE70b] and later greatly simplified by Terlaky [Ter85], using standard convex duality theory (e.g. the convex Farkas theorem).

The aim of this chapter is to derive these results in a completely different setting, using the machinery of conic convex duality described in Chapter 3. This new approach has the advantage of further simplifying the proofs and giving some insight about the reasons why this class of problems has better properties than a general convex problem. We also show that this class of optimization problems can be solved up to a given accuracy in polynomial time, using the theory of self-concordant barriers in the framework of interior-point algorithms (see Chapter 2).

4.1.1 Problem definition

Let us start by introducing the primal l_{p}-norm optimization problem [PE70a, Ter85], which is basically a slight modification of a linear optimization problem where the use of l_{p}-norms applied to linear terms is allowed within the constraints. In order to state its formulation in the most general setting, we need to introduce the following sets: let K = {1, 2, …, r}, I = {1, 2, …, n} and let {I_{k}}_{k∈K} be a partition of I into r classes, i.e. satisfying

∪_{k∈K}I_{k} = I and I_{k} ∩ I_{l} = ∅ for all k ≠ l .

The problem data is given by two matrices A ∈ ℝ^{m×n} and F ∈ ℝ^{m×r} (whose columns will be denoted by a_{i}, i ∈ I and f_{k}, k ∈ K) and four column vectors b ∈ ℝ^{m}, c ∈ ℝ^{n}, d ∈ ℝ^{r} and p ∈ ℝ^{n} such that p_{i} > 1 ∀i ∈ I. Our primal problem consists in optimizing a linear function of a column vector y ∈ ℝ^{m} under a set of constraints involving l_{p}-norms of linear forms, and can be written as

sup b^{T} y s.t. ∑_{i∈I_{k}} \frac{1}{p_{i}} |c_{i} − a_{i}^{T} y|^{p_{i}} ≤ d_{k} − f_{k}^{T} y ∀k ∈ K .  (Pl_{p})

It is readily seen that this formulation is quite general. Indeed,

⋄ linear optimization problems can be modelled by taking n = 0 (and thus I_{k} = ∅ ∀k ∈ K),

which gives

sup b^{T} y s.t. F^{T} y ≤ d ,

⋄ problems of approximation in l_{p}-norm correspond to the case f_{k} = 0 ∀k ∈ K, described in [PE70a, Ter85] and [NN94, Section 6.3.2],

⋄ a convex quadratic constraint can be modelled with a constraint involving an l_{2}-norm. Indeed, \frac{1}{2}y^{T} Qy + f^{T} y + g ≤ 0 (where Q is positive semidefinite) is equivalent to \frac{1}{2} ∥H^{T} y∥^{2} ≤ −f^{T} y − g, where H is a m × s matrix such that Q = HH^{T} (whose columns will be denoted by h_{i}), and can be modelled as

∑_{i=1}^{s} \frac{1}{2} |h_{i}^{T} y|^{2} ≤ −g − f^{T} y ,

4.2 – Cones for l_{p}-norm optimization  75

which has the same form as one constraint of problem (Pl_{p}) with p_{i} = 2 and c_{i} = 0. This implies that linearly and quadratically constrained convex quadratic optimization problems can be modelled as l_{p}-norm optimization problems (since a convex quadratic objective can be modelled using an additional variable, a linear objective and a convex quadratic constraint).

Defining a vector q ∈ ℝ^{n} such that \frac{1}{p_{i}} + \frac{1}{q_{i}} = 1 for all i ∈ I, the dual problem for (Pl_{p}) can be defined as (see e.g. [Ter85])

inf ψ(x, z) = c^{T} x + d^{T} z + ∑_{k∈K|z_{k}>0} z_{k} ∑_{i∈I_{k}} \frac{1}{q_{i}} |\frac{x_{i}}{z_{k}} |^{q_{i}} s.t. \{ \begin{matrix}Ax + F z = b and z ≥ 0 , \\ z_{k} = 0 ⇒ x_{i} = 0 ∀i ∈ I_{k} . \end{matrix} (Dl_{p})

We note that a special convention has been taken to handle the case when one or more components of z are equal to zero: the associated terms are left out of the first sum (to avoid a zero denominator) and the corresponding components of x have to be equal to zero. When compared with the primal problem (Pl_{p}), this problem has a simpler feasible region (mostly defined by linear equalities and nonnegativity constraints) at the price of a highly nonlinear (but convex) objective.

4.1.2 Organization of the chapter

The rest of this chapter is organized as follows. In order to use the setting of conic optimization, we define in Section 4.2 an appropriate convex cone that will allow us to express l_{p}-norm optimization problems as conic programs. We also study some aspects of this cone (closedness, interior, dual). We are then in position to formulate the primal-dual pair (Pl_{p})–(Dl_{p}) using a conic formulation and apply in Section 4.3 the general duality theory for conic optimization, in order to prove the above-mentioned duality results about l_{p}-norm optimization. Section 4.4 deals with algorithmic complexity issues and presents a self-concordant barrier construction for our problem. We conclude with some remarks in Section 4.5.

4.2 Cones for l_{p}-norm optimization

Let us now introduce the L^{p} cone, which will allow us to give a conic formulation of l_{p}-norm optimization problems.

4.2.1 The primal cone

Definition 4.1. Let n ∈ ℕ and p ∈ ℝ^{n} with p_{i} > 1. We define the following set

L^{p} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ_{+} | ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} ≤ κ\}

using in the case of a zero denominator the following convention:

\frac{|x_{i}|}{0} = \begin{cases}+∞ & if x_{i} ≠ 0 , \\ 0 & if x_{i} = 0 . \end{cases}

This convention means that if (x, θ, κ) ∈ L^{p}, θ = 0 implies x = 0^{n}. We start by proving that L^{p} is a convex cone.

Theorem 4.1. L^{p} is a convex cone.

Proof. Let us first introduce the following function

f_{p} : ℝ^{n} × ℝ_{+} ↦ ℝ_{+} ∪ \{+∞\} : (x, θ) ↦ ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} .

With the convention mentioned above, its effective domain is ℝ^{n} × ℝ_{++} ∪ 0^{n} × 0. It is straightforward to check that f_{p} is positively homogeneous, i.e. f_{p}(λx, λθ) = λf_{p}(x, θ) for λ ≥ 0. Moreover, f_{p} is subadditive, i.e. f_{p}(x + x′, θ + θ′) ≤ f_{p}(x, θ) + f_{p}(x′, θ′). In order to show it, we only need to prove the following inequality for all x, x′ ∈ ℝ and θ, θ′ ∈ ℝ_{+}:

\frac{|x|^{p_{i}}}{θ^{p_{i}−1}} + \frac{|x′|^{p_{i}}}{θ′^{p_{i}−1}} ≥ \begin{matrix}|x + x′| & p_{i} \\ (θ + θ′) & p_{i}−1 \end{matrix} .

First observe that this inequality is obviously true if θ or θ′ is equal to 0. When θ and θ′ are both different from 0, we use the well known fact that x^{p_{i}} is a convex function on ℝ_{+} for p_{i} ≥ 1, implying that λa^{p_{i}} +λ′a′^{p_{i}} ≥ (λa + λ′a′)^{p_{i}} for any nonnegative a, a′, λ and λ′ satisfying λ + λ′ = 1. Choosing a = \frac{1}{θ} |x|, a′ = \frac{1}{θ′} |x′|, λ = \frac{θ}{θ+θ′} and λ′ = \frac{θ′}{θ+θ′}, we find that

\frac{θ}{θ + θ′} (\frac{|x|}{θ})^{p_{i}} + \frac{θ′}{θ + θ′} (\frac{|x′|}{θ′})^{p_{i}} ≥ (\frac{θ}{θ + θ′} \frac{|x|}{θ} + \frac{θ′}{θ + θ′} \frac{|x′|}{θ′})^{p_{i}}
\frac{1}{θ + θ′} (\frac{|x|^{p_{i}}}{θ^{p_{i}−1}} + \frac{|x′|^{p_{i}}}{θ′^{p_{i}−1}} ) ≥ (\frac{|x| + |x′|}{θ + θ′})^{p_{i}}
\frac{|x|^{p_{i}}}{θ^{p_{i}−1}} + \frac{|x′|^{p_{i}}}{θ′^{p_{i}−1}} ≥ \begin{matrix}(|x| + |x′|) & p_{i} \\ (θ + θ′) & p_{i}−1 \end{matrix} ≥ \begin{matrix}|x + x′| & p_{i} \\ (θ + θ′) & p_{i}−1 \end{matrix} .

Positive homogeneity and subadditivity imply that f_{p} is a convex function. Since f_{p}(x, θ) ≥ 0 for all x and θ, we notice that L^{p} is the epigraph of f_{p}, i.e.

epi f_{p} = {(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ | f_{p}(x, θ) ≤ κ} = L^{p} .

L^{p} is thus the epigraph of a convex positively homogeneous function, hence a convex cone.

In order to characterize strictly feasible points, we would like to identify the interior of this cone.

Theorem 4.2. The interior of L^{p} is given by

int L^{p} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{++} × ℝ_{++} | ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} < κ\} .

Proof. According to Lemma 7.3 in [Roc70a] we have

int L^{p} = int epi f_{p} = {(x, θ, κ) | (x, θ) ∈ int dom f_{p} and f_{p}(x, θ) < κ} .

The stated result then simply follows from the fact that int dom f_{p} = ℝ^{n} × ℝ_{++}.

Corollary 4.1. The cone L^{p} is solid.

Proof. It suffices to prove that there exists at least one point that belongs to int L^{p}, for example by taking the point (e, 1, n), where e stands for the n-dimensional all-one vector. Indeed, we have ∑_{i=1}^{n} \frac{|1|^{1}}{p_{i}1^{p_{i}−1}} = ∑_{i=1}^{n} \frac{1}{p_{i}} < ∑_{i=1}^{n} 1 = n.

Note 4.1. When n = 0, our cone L^{p} is readily seen to be equivalent to the two-dimensional positive orthant ℝ_{+}^{2}. We also notice that in the special case where p_{i} = 2 for all i, our cone L^{p} becomes

L^{(2,··· ,2)} = {(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ_{+} | ∑_{i=1}^{n} x_{i}^{2} ≤ 2θκ} ,

which is usually called the hyperbolic or rotated second-order cone [LVBL98, Stu99a](it is a simple linear transformation of the usual second-order cone, see Chapter 9).

To illustrate our purpose, we provide in Figure 4.1 the three-dimensional graphs of the boundary surfaces of L^{(5)} and L^{(2)} (corresponding to the case n = 1).

Figure 4.1: The boundary surfaces of L^{(5)} and L^{(2)} (in the case n = 1).

4.2.2 The dual cone

We are now going to determine the dual cone of L^{p}. Let us first recall the following well-known result, known as the weighted arithmetic-geometric inequality.

Lemma 4.1. Let x ∈ ℝ_{++}^{n} and δ ∈ ℝ_{++}^{n} such that ∑_{i=1}^{n} δ_{i} = 1. We have

∏_{i=1}^{n} x_{i}^{δ_{i}} ≤ ∑_{i=1}^{n} δ_{i}x_{i} ,

equality occurring if and only if all x_{i}’s are equal.

This result is easily proved, applying for example Jensen’s inequality [Roc70a, Theorem 4.3] to the convex function x ↦ e^{x}.

We now introduce a useful inequality, which lies at the heart of duality for L^{p} cones [Ter85, NN94]. In order to keep our exposition self-contained, we also include its proof.

Lemma 4.2. Let a, b ∈ ℝ_{+} and α, β ∈ ℝ_{++} such that \frac{1}{α} + \frac{1}{β} = 1. We have the inequality

\frac{a^{α}}{α} + \frac{b^{β}}{β} ≥ ab ,

with equality holding if and only if a^{α} = b^{β}.

Proof. The cases where a = 0 or b = 0 are obvious. When a, b ∈ ℝ_{++}, we can simply apply Lemma 4.1 on a^{α} and b^{β} with weights \frac{1}{α} and \frac{1}{β} (whose sum is equal to one), which gives

\frac{a^{α}}{α} + \frac{b^{β}}{β} ≥ (a^{α})^{1/α}(b^{β})^{1/β} = ab ,

with equality if and only if a^{α} = b^{β}.

For ease of notation, we also introduce the switched cone L_{s}^{p} as the L^{p} cone with its last two components exchanged, i.e.

(x, θ, κ) ∈ L_{s}^{p} ⇔ (x, κ, θ) ∈ L^{p} .

We are now ready to describe the dual of L^{p}.

Theorem 4.3 (Dual of L^{p}). Let p, q ∈ ℝ_{++}^{n} such that \frac{1}{p_{i}} + \frac{1}{q_{i}} = 1 for each i. The dual of L^{p} is L_{s}^{q}.

Proof. By definition of the dual cone, we have

(L^{p})^{∗} = {v^{∗} ∈ ℝ^{n} × ℝ × ℝ | v^{T} v^{∗} ≥ 0 for all v ∈ L^{p}} .

We start by showing that L_{s}^{q} ⊆ (L^{p})^{∗}.

Let v^{∗} = (x^{∗}, θ^{∗}, κ^{∗}) ∈ L_{s}^{q} and v = (x, θ, κ) ∈ L^{p}. We are going to prove that v^{T} v^{∗} ≥ 0, which will imply the desired inclusion. The case when θ = 0 is easily handled: we have then x = 0 implying v^{T} v^{∗} = κκ^{∗} ≥ 0. Similarly we can eliminate the case where κ^{∗} = 0. In the remaining cases, we use the definitions of L^{p} and L_{s}^{q} to get

f_{p}(x, θ) = ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} ≤ κ and f_{q}(x^{∗}, κ^{∗}) = ∑_{i=1}^{n} \frac{|x_{i}^{∗}|^{q_{i}}}{q_{i}κ^{∗}^{q_{i}−1}} ≤ θ^{∗} .

Dividing respectively by θ and κ^{∗} and adding the resulting inequalities we find

∑_{i=1}^{n}(\frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}}} + \frac{|x_{i}^{∗}|^{q_{i}}}{q_{i}κ^{∗}^{q_{i}}} ) ≤ \frac{κ}{θ} + \frac{θ^{∗}}{κ^{∗}} .  (4.1)

Applying now Lemma 4.2 to each pair \frac{|x_{i}|}{θ}, \frac{|x_{i}^{∗}|}{κ^{∗}} we get

∑_{i=1}^{n} \frac{|x_{i}|}{θ} \frac{|x_{i}^{∗}|}{κ^{∗}} ≤ \frac{κ}{θ} + \frac{θ^{∗}}{κ^{∗}} ,  (4.2)

which is equivalent to

∑_{i=1}^{n} |x_{i}| |x_{i}^{∗}| ≤ κκ^{∗} + θθ^{∗} .

Finally, noting that x_{i}x_{i}^{∗} ≥ − |x_{i}| |x_{i}^{∗}| we conclude that

v^{T} v^{∗} = x^{T} x^{∗} + κκ^{∗} + θθ^{∗} = ∑_{i=1}^{n} x_{i}x_{i}^{∗} + κκ^{∗} + θθ^{∗} ≥ ∑_{i=1}^{n} − |x_{i}| |x_{i}^{∗}| + κκ^{∗} + θθ^{∗} ≥ 0 , (4.3)

showing that L_{s}^{q} ⊆ (L^{p})^{∗}.

Let us prove now the reverse inclusion, i.e. (L^{p})^{∗} ⊆ L_{s}^{q}.

Let v^{∗} = (x^{∗}, θ^{∗}, κ^{∗}) ∈ (L^{p})^{∗}. We have to show that v^{∗} ∈ L_{s}^{q}, using that v^{T} v^{∗} ≥ 0 for every v = (x, θ, κ) ∈ L^{p}. Choosing v = (0, 0, 1), we first ensure that v^{T} v^{∗} = κ^{∗} ≥ 0. We distinguish the cases κ^{∗} = 0 and κ^{∗} > 0. If κ^{∗} = 0, we have that v^{T} v^{∗} = x^{T} x^{∗} + θθ^{∗} ≥ 0 for every v = (x, θ, κ) ∈ L^{p}. Choosing θ = 1 and κ ≥ f_{p}(x, 1) for any x ∈ ℝ^{n}, we find that x^{T} x^{∗} + θ^{∗} ≥ 0 for all x ∈ ℝ^{n}, which implies x^{∗} = 0 and θ^{∗} ≥ 0 and thus v^{∗} ∈ L_{s}^{q}. When κ^{∗} > 0, we can always choose a v ∈ L^{p} such that

\frac{|x_{i}|^{p_{i}}}{θ^{p_{i}}} = \frac{|x_{i}^{∗}|^{q_{i}}}{κ^{∗}^{q_{i}}} , x_{i}x_{i}^{∗} ≤ 0 and f_{p}(x, θ) = ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} = κ .  (4.4)

Writing

0 ≤ \frac{v^{T} v^{∗}}{θκ^{∗}} = (\frac{x}{θ})^{T} (\frac{x^{∗}}{κ^{∗}} )^{T} + \frac{θ^{∗}}{κ^{∗}} + \frac{κ}{θ}

= ∑_{i=1}^{n} \frac{x_{i}}{θ} \frac{x_{i}^{∗}}{κ^{∗}} + \frac{θ^{∗}}{κ^{∗}} + \frac{κ}{θ}

= ∑_{i=1}^{n} −\frac{|x_{i}|}{θ} \frac{|x_{i}^{∗}|}{κ^{∗}} + \frac{θ^{∗}}{κ^{∗}} + \frac{κ}{θ} ,

using the case of equality of Lemma 4.2 on the pairs \frac{|x_{i}|}{θ}, \frac{|x_{i}^{∗}|}{κ^{∗}} and the choice of v in (4.4),

= − ∑_{i=1}^{n}(\frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}}} + \frac{|x_{i}^{∗}|^{q_{i}}}{q_{i}κ^{∗}^{q_{i}}} ) + \frac{θ^{∗}}{κ^{∗}} + \frac{κ}{θ}

= \frac{θ^{∗}}{κ^{∗}} − ∑_{i=1}^{n} \frac{|x_{i}^{∗}|^{q_{i}}}{q_{i}κ^{∗}^{q_{i}}} ,

and finally multiplying by κ^{∗} leads to

∑_{i=1}^{n} \frac{|x_{i}^{∗}|^{q_{i}}}{q_{i}κ^{∗}^{q_{i}−1}} ≤ θ^{∗} ,

i.e. v^{∗} ∈ L_{s}^{q}, showing that (L^{p})^{∗} ⊆ L_{s}^{q} and thus (L^{p})^{∗} = L_{s}^{q}.

The dual of a L^{p} cone is thus equal, up to a permutation of two variables, to another L^{p} cone with a dual vector of exponents.

Corollary 4.2. We also have (L_{s}^{p})^{∗} = L^{q}, (L^{q})^{∗} = L_{s}^{p} and (L_{s}^{q})^{∗} = L^{p}.

Proof. Obvious considering both the symmetry between L^{p} and L_{s}^{q} and the symmetry between p and q.

Corollary 4.3. L^{p} and L_{s}^{q} are solid and pointed.

Proof. We have already proved that L^{p} is solid which, for obvious symmetry reasons, implies that its switched counterpart L_{s}^{q} is also solid. Since pointedness is the property that is dual to solidness (Theorem 3.3), noting that L^{p} = (L_{s}^{q})^{∗} and L_{s}^{q} = (L^{p})^{∗} is enough to prove that L^{p} and L_{s}^{q} are also pointed.

Corollary 4.4. L^{p} and L_{s}^{q} are closed.

Proof. Starting with (L^{p})^{∗} = L_{s}^{q} and taking the dual of both sides, we find ((L^{p})^{∗})^{∗} = (L_{s}^{q})^{∗}. Since (L_{s}^{q})^{∗} = L^{p} by Corollary 4.2 and ((L^{p})^{∗})^{∗} = cl L^{p} [Roc70a, page 121], we have cl L^{p} = L^{p}, hence L^{p} is closed. The switched cone L_{s}^{q} is obviously closed as well.

We can also provide a direct proof of the closedness of L^{p}: using the fact that it is the epigraph of f_{p}, it is enough to show that f_{p} is a lower semicontinuous function [Roc70a, Theorem 7.1]. Being convex, f_{p} is continuous on the interior of its effective domain, i.e. when θ > 0. When θ = 0, we have to prove that

lim_{(x,θ)→(x^{0},0^{+})} f_{p}(x, θ) ≥ f_{p}(x^{0}, 0) .

On the one hand, if x_{i}^{0} ≠ 0 for some index i, we have that f_{p}(x^{0}, 0) = +∞ but also that lim_{(x,θ)→(x^{0},0^{+})} f_{p}(x, θ) = +∞, since the term \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} tends to +∞ when (x_{i}, θ) tends to (x_{i}^{0}, 0), hence the inequality is true. On the other hand, if x^{0} = 0, we have to check that lim_{(x,θ)→(0,0^{+})} f_{p}(x, θ) ≥ f_{p}(0, 0) = 0, which is obviously also true. From this we can conclude that f_{p} is lower semicontinuous and hence L^{p} is closed.

Note however that f_{p} is not continuous in (0, 0). Choosing an arbitrary positive constant M and defining for example x_{i}(θ) = (Mp_{i})^{1/p_{i}}θ^{1/q_{i}}, so that x(θ) → 0 when θ → 0^{+}, we have that lim_{θ→0^{+}} f(x(θ), θ) = nM ≠ f(0, 0) = 0. The limit of f_{p} at (0, 0) can indeed take any positive value^{1}.

1However, taking x(θ) proportional to θ, namely x_{i}(θ) = L_{i}θ, we have lim_{θ→0^{+}} f(x(θ), θ) = f(0, 0) = 0, i.e.

f_{p} is continuous on its restrictions to lines passing through the origin.

Note 4.2. As special cases, we note that when n = 0, (L^{p})^{∗} is equivalent to ℝ_{+}^{2}, which is the usual dual for L^{p} = ℝ_{+}^{2}. In the case of p_{i} = 2 ∀i, we find

(L^{(2,··· ,2)})^{∗} = L_{s}^{(2,··· ,2)} = {(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ_{+} | ∑_{i=1}^{n} x_{i}^{2} ≤ 2θκ} ,

which is the expected result. Note that apart from these two special cases, L^{p} is in general not self-dual.

Note 4.3 (Self-duality of L^{p} cones with n = 1). Let us examine the special case of threedimensional L^{p} cones, i.e. assume n = 1. Figure 4.2, representing L^{( \frac{5}{4} )}, illustrates our point: up to a permutation of variables, it is equal to (L^{(5)})^{∗} (since 1/5 + 1/\frac{5}{4} = 1) and is different from L^{(5)}, and hence these cones are not self-dual. However, in the particular case where n = 1, this difference is not as great as it could be. Namely, one can show easily that L^{(p)} and its dual are equal up to a simple scaling of some of the variables. Indeed, we have

(x, θ, κ) ∈ L^{(p)} ⇔ |x|^{p} ≤ pκθ^{p−1}
⇔ |x|^{q} ≤ p^{\frac{q}{p}} κ^{\frac{q}{p}} θ^{(p−1) \frac{q}{p}}

using \frac{q}{p} = q \frac{1}{p} = q(1 − \frac{1}{q}) = q − 1 and (p − 1)\frac{q}{p} = (1 − \frac{1}{p})q = \frac{1}{q}q = 1

⇔ |x|^{q} ≤ p^{q−1}κ^{q−1}θ
⇔ |x|^{q} ≤ q(pκ)^{q−1} \frac{θ}{q}
⇔ (x, \frac{θ}{q} , pκ) ∈ L_{s}^{(q)} = (L^{(p)})^{∗} .

From another point of view, we could also state that these two cones are self-dual with respect to a modified inner product that takes this scaling of the variables into account.

Figure 4.2: The boundary surfaces of L^{( \frac{5}{4} )} and L^{(5)} (in the case n = 1).

Our last theorem in this section describes the cases where two vectors from L^{p} and L_{s}^{q} are orthogonal to each other, which will be used in the study of the duality properties.

Theorem 4.4 (orthogonality conditions). Let v = (x, θ, κ) ∈ L^{p} and v^{∗} = (x^{∗}, θ^{∗}, κ^{∗}) ∈ L_{s}^{q}. We have v^{T} v^{∗} = 0 if and only if the following set of conditions holds

κ^{∗}(f_{p}(x, θ) − κ) = 0  (4.5a)
θ(f_{q}(x^{∗}, κ^{∗}) − θ^{∗}) = 0  (4.5b)
κ^{∗} \frac{|x_{i}|^{p_{i}}}{θ^{p_{i}−1}} = θ \frac{|x_{i}^{∗}|^{q_{i}}}{κ^{∗}^{q_{i}−1}}  (4.5c)
x_{i}x_{i}^{∗} ≤ 0 for all i .  (4.5d)

Proof. When θ > 0 and κ^{∗} > 0, a careful reading of the first part of the proof of Theorem 4.3 shows that equality occurs if and only if all conditions in (4.5) are fulfilled. Namely, (4.5a) and (4.5b) are responsible for equality in (4.1), (4.5c) ensures that we are in the case of equality of Lemma 4.2 for inequality (4.2) and the last condition (4.5d) is necessary for equality in (4.3).

When θ = 0 but κ^{∗} > 0, we have x = 0 and thus v^{T} v^{∗} = κκ^{∗}. This quantity is zero if and only if κ = 0, which is equivalent in this case to f_{p}(x, θ) = κ and occurs if and only if (4.5a) is satisfied (all the other conditions being trivially fulfilled). A similar reasoning takes care of the case θ > 0, κ^{∗} = 0.

Finally, when θ = κ^{∗} = 0, we have x = x^{∗} = 0 and v^{T} v^{∗} = 0, while the set of conditions (4.5) is also always satisfied.

4.3 Duality for l_{p}-norm optimization

This is the main section , where we show how a primal-dual pair of l_{p}-norm optimization problems can be modelled using the L^{p} and L_{s}^{q} cones and how this allows us to derive the relevant duality properties.

4.3.1 Conic formulation

Let us restate here for convenience the definition of the standard primal l_{p}-norm optimization problem (Pl_{p}).

sup b^{T} y s.t. ∑_{i∈I_{k}} \frac{1}{p_{i}} |c_{i} − a_{i}^{T} y|^{p_{i}} ≤ d_{k} − f_{k}^{T} y ∀k ∈ K  (Pl_{p})

(where K = {1, 2, …, r}, I = {1, 2, …, n}, {I_{k}}_{k∈K} is a partition of I into r classes, A ∈ ℝ^{m×n} and F ∈ ℝ^{m×r} (whose columns will be denoted by a_{i}, i ∈ I and f_{k}, k ∈ K), y ∈ ℝ^{m}, b ∈ ℝ^{m}, c ∈ ℝ^{n}, d ∈ ℝ^{r} and p ∈ ℝ^{n} such that p_{i} > 1 ∀i ∈ I).

Let us now model problem (Pl_{p}) with a conic formulation. The following notation will be useful in this context: v_{S} (resp. M_{S}) denotes the restriction of column vector v (resp. matrix M) to the components (resp. rows) whose indices belong to set S.

4.3 – Duality for l_{p}-norm optimization  83

We start by introducing an auxiliary vector of variables x^{∗} ∈ ℝ^{n} to represent the argument of the power functions, namely we let

x_{i}^{∗} = c_{i} − a_{i}^{T} y for all i ∈ I or, in matrix form, x^{∗} = c − A^{T} y ,

and we also need additional variables z^{∗} ∈ ℝ^{r} for the linear term forming the right-hand side of the inequalities

z_{k}^{∗} = d_{k} − f_{k}^{T} y for all k ∈ K or, in matrix form, z^{∗} = d − F^{T} y .

Our problem is now equivalent to

sup b^{T} y s.t. A^{T} y + x^{∗} = c, F^{T} y + z^{∗} = d and ∑_{i∈I_{k}} \frac{1}{p_{i}} |x_{i}^{∗}|^{p_{i}} ≤ z_{k}^{∗} ∀k ∈ K ,

where we can easily plug our definition of the L^{p} cone, provided we fix variables θ to 1

sup b^{T} y s.t. A^{T} y + x^{∗} = c, F^{T} y + z^{∗} = d and (x_{I_{k}}^{∗}, 1, z_{k}^{∗}) ∈ L^{p^{k}} ∀k ∈ K

(where for convenience we defined vectors p^{k} = (p_{i} | i ∈ I_{k}) for k ∈ K). We finally introduce an additional vector of fictitious variables v^{∗} ∈ ℝ^{r} whose components are fixed to 1 by linear constraints to find

sup b^{T} y s.t. A^{T} y + x^{∗} = c, F^{T} y + z^{∗} = d, v^{∗} = e and (x_{I_{k}}^{∗}, v_{k}^{∗}, z_{k}^{∗}) ∈ L^{p^{k}} ∀k ∈ K

(where e stands again for the all-one vector). Rewriting the linear constraints with a single matrix equality, we end up with

sup b^{T} y s.t. (\begin{matrix}A^{T} \\ F^{T} \\ 0 \end{matrix} ) y + (\begin{matrix}x^{∗} \\ z^{∗} \\ v^{∗} \end{matrix}) = (\begin{matrix}c \\ d \\ e \end{matrix}) and (x_{I_{k}}^{∗}, v_{k}^{∗}, z_{k}^{∗}) ∈ L^{p^{k}} ∀k ∈ K ,  (CPl_{p})

which is exactly a conic optimization problem in the dual^{2} form (CD), using variables (ỹ, s̃), data (Ã, b̃, c̃) and a cone C^{∗} such that

ỹ = y, s̃ = (\begin{matrix}x^{∗} \\ z^{∗} \\ v^{∗} \end{matrix}) , Ã = (A F 0) , b̃ = b, c̃ = (\begin{matrix}c \\ d \\ e \end{matrix}) and C^{∗} = L^{p^{1}} × L^{p^{2}} × ⋯ × L^{p^{r}} ,

where C^{∗} has been defined according to Note 3.1, since we have to deal with multiple conic constraints involving disjoint sets of variables.

Using properties of L^{p} proved in the previous section, it is straightforward to show that C^{∗} is a solid, pointed, closed convex cone whose dual is

(C^{∗})^{∗} = C = L_{s}^{q^{1}} × L_{s}^{q^{2}} × ⋯ × L_{s}^{q^{r}} ,

another solid, pointed, closed convex cone (where we have defined a vector q ∈ ℝ^{n} such that \frac{1}{p_{i}} + \frac{1}{q_{i}} = 1 for all i ∈ I and vectors q^{k} such that q^{k} = (q_{i} | i ∈ I_{k}) for k ∈ K). This allows us to derive a dual problem to (CPl_{p}) in a completely mechanical way and find the following conic optimization problem, expressed in the primal form (CP) (since the dual of a problem in dual form is a problem in primal form):

2This is the reason why we added a ^{∗} superscript to the notation of our additional variables, in order to emphasize the fact that the primal l_{p}-norm optimization problem (Pl_{p}) is in fact in the dual conic form (CD).

inf (c^{T} d^{T} e^{T} ) (\begin{matrix}x \\ z \\ v \end{matrix}) s.t. (A F 0) (\begin{matrix}x \\ z \\ v \end{matrix}) = b and (x_{I_{k}}, v_{k}, z_{k}) ∈ L_{s}^{q^{k}} for all k ∈ K ,

which is equivalent to

inf c^{T} x + d^{T} z + e^{T} v s.t. Ax + F z = b and (x_{I_{k}}, v_{k}, z_{k}) ∈ L_{s}^{q^{k}} for all k ∈ K , (CDl_{p})

where x ∈ ℝ^{n}, z ∈ ℝ^{r} and v ∈ ℝ^{r} are the dual variables we optimize. This problem can be simplified: making the conic constraints explicit, we find

inf c^{T} x + d^{T} z + e^{T} v s.t. Ax + F z = b, ∑_{i∈I_{k}} \frac{|x_{i}|^{q_{i}}}{q_{i}z_{k}^{q_{i}−1}} ≤ v_{k} ∀k ∈ K and z ≥ 0 ,

keeping in mind the convention on zero denominators that in effect implies z_{k} = 0 ⇒ x_{I_{k}} = 0. Finally, we can remove the v variables from the formulation since they are only constrained by the sum inequalities, which have to be tight at any optimal solution. We can thus directly incorporate these sums into the objective function, which leads to

inf ψ(x, z) = c^{T} x + d^{T} z + ∑_{k∈K|z_{k}>0} z_{k} ∑_{i∈I_{k}} \frac{1}{q_{i}} |\frac{x_{i}}{z_{k}} |^{q_{i}} s.t. \{ \begin{matrix}Ax + F z = b and z ≥ 0 , \\ z_{k} = 0 ⇒ x_{i} = 0 ∀i ∈ I_{k} . \end{matrix} (Dl_{p})

Unsurprisingly, the dual formulation (Dl_{p}) we have just found without much effort is exactly the standard form for a dual l_{p}-norm optimization problem [Ter85].

4.3.2 Duality properties

We are now able to prove the weak duality property for the l_{p}-norm optimization problem.

Theorem 4.5 (Weak duality). If y is feasible for (Pl_{p}) and (x, z) is feasible for (Dl_{p}), we have ψ(x, z) ≥ b^{T} y. Equality occurs if and only if for all k ∈ K and i ∈ I_{k}

z_{k}(∑_{i∈I_{k}} \frac{1}{p_{i}} |c_{i} − a_{i}^{T} y|^{p_{i}} + f_{k}^{T} y − d_{k}) = 0, x_{i}(c_{i} − a_{i}^{T} y) ≤ 0, z_{k} |c_{i} − a_{i}^{T} y|^{p_{i}} = \frac{|x_{i}|^{q_{i}}}{z_{k}^{q_{i}−1}} . (4.6)

Proof. Let y and (x, z) be feasible for (Pl_{p}) and (Dl_{p}). Choosing v_{k} = f_{q^{k}}(x_{I_{k}}, z_{k}) for all k ∈ K, we have that (x, z, v) is feasible for (CDl_{p}) with the same objective function, i.e. with c^{T} x + d^{T} z + e^{T} v = ψ(x, z). Moreover, computing (x^{∗}, z^{∗}, v^{∗}) from y in order to satisfy the linear constraints in (CPl_{p}), i.e. according to

x_{i}^{∗} = c_{i} − a_{i}^{T} y, z_{k}^{∗} = d_{k} − f_{k}^{T} y, v_{k}^{∗} = 1 ,  (4.7)

we have that (x^{∗}, z^{∗}, v^{∗}, y) is feasible for (CPl_{p}). The standard weak duality property for the conic pair (CPl_{p})–(CDl_{p}) from Theorem 3.4 then states that c^{T} x + d^{T} z + e^{T} v ≥ b^{T} y, which in turn implies ψ(x, z) ≥ b^{T} y.

We proceed now to investigate the equality conditions. At the optimum, variables v_{k} must assume their lower bounds so that we can still assume that v_{k} = f_{q^{k}}(x_{I_{k}}, z_{k}) holds for all k ∈ K. We also keep variables (x^{∗}, z^{∗}, v^{∗}) defined by (4.7). From the weak duality Theorem 3.4, we know that equality can only occur if the primal and dual vectors of variables are orthogonal to each other for each conic constraint, i.e. (x_{I_{k}}^{∗}, z_{k}^{∗}, v_{k}^{∗})^{T} (x_{I_{k}}, z_{k}, v_{k}) = 0 for all k ∈ K.

Having (x_{I_{k}}^{∗}, v_{k}^{∗}, z_{k}^{∗})^{T} ∈ L^{p^{k}} and (x_{I_{k}}, v_{k}, z_{k}) ∈ L_{s}^{q^{k}}, Theorem 4.4 gives us the necessary and sufficient conditions for equality to happen

z_{k}(f_{p^{k}}(x_{I_{k}}^{∗}, v_{k}^{∗}) − z_{k}^{∗}) = 0, v_{k}^{∗}(f_{q^{k}}(x_{I_{k}}, z_{k}) − v_{k}) = 0, z_{k} \frac{|x_{i}^{∗}|^{p_{i}}}{v_{k}^{∗}^{p_{i}−1}} = v_{k}^{∗} \frac{|x_{i}|^{q_{i}}}{z_{k}^{q_{i}−1}} , x_{i}x_{i}^{∗} ≤ 0 (4.8)

for all i ∈ I_{k} and k ∈ K. The second condition is always satisfied while the other three conditions can be readily simplified using (4.7) to give the announced inequalities (4.6).

The weak duality property is a rather straightforward consequence of the convexity of the problems, and in fact can be proved without too many difficulties without sophisticated tools from duality theory. However, this is not the case with the next theorem, which deals with a strong duality property.

In the case of a general pair of primal and dual conic problems, the duality gap at the optimum is not always equal to zero, neither are the primal or dual optimum objective values always attained by feasible solutions (see the examples in Section3.3). However, it is wellknown that in the special case of linear optimization, we always have a zero duality gap and attainment of both optimum objective values. The status of l_{p}-norm optimization lies somewhere between these two situations: the duality gap is always equal zero but attainment of the optimum objective value can only be guaranteed for the primal problem.

In the course of our proof, we will need to use the well-known Goldman-Tucker theorem [GT56] for linear optimization, which we state here for reference.

Theorem 4.6 (Goldman-Tucker). Let us consider the following primal-dual pair of linear optimization problems in standard form:

min c^{T} x s.t. Ax = b and x ≥ 0 and max b^{T} y s.t. A^{T} y + s = c and s ≥ 0 .

If both problems are feasible, there exists a unique partition (B, N) of the index set common to vectors x and s such that

⋄ every optimal solution x̂ to the primal problem satisfies x̂_{N} = 0.

⋄ every optimal solution (ŷ, ŝ) to the dual problem satisfies ŝ_{B} = 0.

This partition is called the optimal partition. Moreover, there exists at least an optimal primal-dual solution (x̂, ŷ, ŝ) such that x̂ + ŝ > 0, hence satisfying x̂_{B} > 0 and ŝ_{N} > 0. Such a pair is called a strictly complementary pair^{3}.

3This optimal partition can be computed in polynomial time by interior-point methods. Indeed, it is possible to prove for example that the short-step algorithm presented in Chapter 2 converges to a strictly complementary solution, and thus allows us to identify the optimal partition unequivocally.

This theorem is central to the theory of duality for linear optimization. Its most important consequence is the fact that any pair of primal-dual optimal solutions x̂ and (ŷ, ŝ) must have a zero duality gap. Indeed, the duality gap is equal to x̂^{T} ŝ (see Theorem 3.4) and the theorem implies that x̂_{N} = 0 and ŝ_{B} = 0, which leads to

x̂^{T} ŝ = ∑_{i∈B} x̂_{i}ŝ_{i} + ∑_{i∈N} x̂_{i}ŝ_{i} = 0

since (B, N) is a partition of the index set of the variables. One can also consider this theorem as a version of the strong duality Theorem 3.5 specialized for linear optimization, with the important difference that it is valid even when no Slater point exists.

The strong duality theorem for l_{p}-norm optimization we are about to prove is the following:

Theorem 4.7 (Strong duality). If both problems (Pl_{p}) and (Dl_{p}) are feasible, the primal optimal objective value is attained with a zero duality gap, i.e.

p^{∗} = max b^{T} y s.t. ∑_{i∈I_{k}} \frac{1}{p_{i}} |c_{i} − a_{i}^{T} y|^{p_{i}} ≤ d_{k} − f_{k}^{T} y ∀k ∈ K

= inf ψ(x, z) s.t. \{ \begin{matrix}Ax + F z = b and z ≥ 0 \\ z_{k} = 0 ⇒ x_{i} = 0 ∀i ∈ I_{k} \end{matrix} = d^{∗} .

Proof. The strong duality Theorem 3.5 tells us that zero duality gap and primal attainment are guaranteed by the existence of a strictly interior dual feasible solution (excluding the case of an unbounded dual). Let (x, z) be a feasible solution for (Dl_{p}). We would like to complement it with a vector v such that the corresponding solution (x, z, v) is strictly feasible for the conic formulation (CDl_{p}).

Since cone C is the cartesian products of the set of cones L_{s}^{q^{k}} for k ∈ K, we need in fact for (x, z, v) to be a strictly feasible solution of (CDl_{p}) that (x_{I_{k}}, z_{k}, v_{k}) ∈ int L_{s}^{q^{k}} holds for all k ∈ K. Using now Theorem 4.2 to identify the interior of the L_{s}^{q} cones, we see that both conditions v_{k} > f_{p^{k}}(x_{I_{k}}, z_{k}) and z_{k} > 0 have to be valid for all k ∈ K.

Since vector v contains only free variables and is not constrained by the linear constraints, it is always possible to choose it such that v_{k} > f_{p^{k}}(x_{I_{k}}, z_{k}) for all k ∈ K. However, the situation is much different for z: it is unfortunately not always possible to find a strictly positive z, since it may happen that the linear constraints combined with the nonnegativity constraint on z force one or more of the components z_{k} to be equal to zero for all primal feasible solutions. Here is an outline of the three-step strategy we are going to follow:

a. Since some components of z may prevent the existence of a strictly feasible solution to (CDl_{p}), we are going to define a restricted version of (CDl_{p}) where those problematic components of z and the associated variables x have been removed. Hopefully, this restricted problem (RDl_{p}) will not behave too differently from the original because the zero components of z and x did not play a crucial role in it. b. Since this restricted problem will now admit a strictly feasible solution, its dual problem

(RPl_{p}) (which is a problem in primal form) has a duality gap equal to zero with its optimal objective value attained by some solution.

c. The last step of our proof will be to convert this optimal solution with a zero duality gap for the restricted primal problem (RPl_{p}) into an optimal solution for the original primal problem (CPl_{p}).

The whole procedure can be summarized with the following diagram:

(Pl_{p}) ≡ (CPl_{p})  ⟷^{Weak}  (CDl_{p})  ≡ (Dl_{p})
c. ↕  ↕ a.
(RPl_{p})

Strong (zero gap)

⟷

(RDl_{p})
↓

(Attainment) b.

↑

(Strictly feasible)

Let us first identify the problematic z_{k}’s that are identically equal to zero for all feasible solutions. This can be done by solving the following linear optimization problem:

min 0 s.t. Ax + F z = b and z ≥ 0 .  (ALP)

This problem has the same feasible region as our dual problem (Dl_{p}) (actually, its feasible region can be slightly larger from the point of view of the x variables, since the special constraints z_{k} = 0 ⇒ x_{I_{k}} = 0 have been omitted, but this does not have any effect on our reasoning). We are thus looking for components of z that are equal to zero on the whole feasible region of (ALP).

Since this problem has a zero objective function, all its feasible solutions are optimal and we can therefore deduce that if a variable z_{k} is zero for all feasible solutions to problem (ALP), it is zero for all optimal solution to problem (ALP). In order to use the Goldman-Tucker theorem, we also write the dual^{4} of problem (ALP):

max b^{T} y s.t. A^{T} y = 0, F^{T} y + z^{∗} = 0 and z^{∗} ≥ 0 .  (ALD)

Both (ALP) and (ALD) are feasible (the former because (Dl_{p}) is assumed to be feasible, the latter because (y, z^{∗}) = (0, 0) is always a feasible solution), which means that the Goldman-Tucker theorem is applicable. Having now the optimal partition (B, N) at hand, we observe that the index set N defines exactly the set of variables z_{i} that are identically zero on the feasible regions of problems (ALP) and (Dl_{p}). We are thus now ready the apply the strategy outlined above.

a. Let us introduce the reduced primal-dual pair of l_{p}-norm optimization problems where variables z_{k} and x_{I_{k}} with k ∈ N have been removed. We start with the dual problem

inf c_{I_{B}}^{T}x_{I_{B}} +d_{B}^{T}z_{B} +e_{B}^{T}v_{B} s.t. A_{I_{B}}x_{I_{B}} +F_{B}z_{B} = b, (x_{I_{k}}, v_{k}, z_{k}) ∈ L_{s}^{q^{k}} ∀k ∈ B , (RDl_{p})

4Although problem (ALP) is not exactly formulated in the standard form used to state Theorem 4.6, the same results hold in the case of a general linear optimization problem.

where I_{B} stands for ∪_{k∈B}I_{k}. It is straightforward to check that this problem is completely equivalent to problem (CDl_{p}), since the variables z_{N} and x_{I_{N}} we removed, being forced to zero for all feasible solutions, had no contribution to the objective or to the linear constraints in (CDl_{p}).

The corresponding conic constraints become (0, v_{k}, 0) ∈ L_{s}^{q^{k}} ⇔ v_{k} ≥ 0 ∀k ∈ N, which imply at the optimum that v_{k} = 0 ∀k ∈ N, showing that variables v_{N} can also be safely removed without changing the optimum objective value. We can thus conclude that inf (RDl_{p}) = inf (CDl_{p}) = inf (Dl_{p}).

b. Because of the second part of the Goldman-Tucker theorem, there is at least one feasible solution to (ALP) such that z_{B} > 0. Combining the (x_{I_{B}}, z_{B}) part of this solution with a vector v_{B} with sufficiently large components gives us a strictly feasible solution for (RDl_{p}) (z_{k} > 0 and v_{k} > f_{q^{k}}(x_{I_{k}}, z_{k}) for all k ∈ B), which is exactly what we need to apply our strong duality Theorem 3.5. Let us first write down the dual problem of (RDl_{p}), the restricted primal:

sup b^{T} y s.t. \{ \begin{matrix}A_{I_{B}}^{T}y + x_{I_{B}}^{∗} = c_{I_{B}}, F_{B}^{T} y + z_{B}^{∗} = d_{B}, v_{B}^{∗} = e, \\ (x_{I_{k}}^{∗}, v_{k}^{∗}, z_{k}^{∗}) ∈ L^{p^{k}} ∀k ∈ B . \end{matrix}  (RPl_{p})

We cannot be in the first case of the strong duality Theorem 3.5, since unboundedness of (RDl_{p}) would imply unboundedness of the original problem (Dl_{p}) which in turn would prevent the existence of a feasible primal solution (simple consequence of the weak duality theorem). We can thus conclude that there exists an optimal solution to (RPl_{p}) (x̂_{I_{B}}^{∗}, ẑ_{B}^{∗}, v̂_{B}^{∗}, ŷ) such that b^{T} ŷ = max (RPl_{p}) = inf (RDl_{p}).

c. Combining the results obtained so far, we have proved that max (RPl_{p}) = inf (Dl_{p}). The last step we need to perform is to prove that max (Pl_{p}) = max (RPl_{p}), i.e. that the optimum objective of (Pl_{p}) is attained and that it is equal to the optimal objective value of (RPl_{p}). Unfortunately, the apparently most straightforward way to do this, namely using the optimal solution ŷ we have at hand for problem (RPl_{p}), does not work since it is not necessarily feasible for problem (CPl_{p}). The reason is that (CPl_{p}) contains additional conic constraints (the ones corresponding to k ∈ N) which are not guaranteed to be satisfied by the optimal solution ŷ of the restricted problem. We can however overcome this difficulty by perturbing this solution by a suitably chosen vector such that

⋄ feasibility for the constraints k ∈ B is not lost,

⋄ feasibility for the constraints k ∈ N can be gained.

Let us consider (x̄, z̄, ȳ, z̄^{∗}), a strictly complementary solution to the primal-dual pair (ALP)–(ALD) whose existence is guaranteed by the Goldman-Tucker theorem. We have thus z̄_{N}^{∗} > 0 and z̄_{B}^{∗} = 0. Since all primal solutions have a zero objective, the optimal dual objective value also satisfies b^{T} ȳ = 0. Summarizing the properties of ȳ obtained so far, we can write

b^{T} ȳ = 0, A^{T} ȳ = 0, F_{B}^{T} ȳ = −z̄_{B}^{∗} = 0 and F_{N}^{T} ȳ = −z̄_{N}^{∗} < 0 .

Let us now consider y = ŷ + λȳ with λ ≥ 0 as a solution of (CPl_{p}) and compute the value of x^{∗} and z^{∗} given by (4.7), distinguishing the B and N parts (we already know that v^{∗} = e):

x_{I_{B}}^{∗} = c_{I_{B}} − A_{I_{B}}^{T}y = c_{I_{B}} − A_{I_{B}}^{T} ŷ  = x̂_{I_{B}}^{∗}  (using A_{I_{B}}^{T} ȳ = 0)
z_{B}^{∗} = d_{B} − F_{B}^{T} y = d_{B} − F_{B}^{T} ŷ  = ẑ_{B}^{∗}  (using F_{B}^{T} ȳ = 0)
x_{I_{N}}^{∗} = c_{I_{N}} − A_{I_{N}}^{T} y = c_{I_{N}} − A_{I_{N}}^{T} ŷ  = x̂_{I_{N}}^{∗}  (using A_{I_{N}}^{T} ȳ = 0)
z_{N}^{∗} = d_{N} − F_{N}^{T} y = d_{N} − F_{N}^{T} ŷ + λz̄_{N}^{∗}  (using − F_{N}^{T} ȳ = z̄_{N}^{∗}) .

The conic constraints corresponding to k ∈ B remain valid for all λ, since the associated variables do not vary with λ. Considering now the constraints for k ∈ N, we see that x_{I_{N}}^{∗} does not depend on λ, while z_{N}^{∗} can be made arbitrarily large by increasing λ, due

to the fact that z̄_{N}^{∗} > 0. Choosing a sufficiently large λ, we can force (x_{I_{k}}^{∗}, 1, z_{k}^{∗}) ∈ L_{s}^{q^{k}} for k ∈ N and thus make (x^{∗}, v^{∗}, z^{∗}, y) feasible for (CPl_{p}). Obviously, we also have that y is feasible for (Pl_{p}) with the same objective value.

Evaluating this objective value, we find that b^{T} y = b^{T} ŷ+λb^{T} ȳ = b^{T} ŷ = max (RPl_{p}), i.e. the feasible solution y we constructed has the same objective value for (CPl_{p}) and (Pl_{p}) as ŷ for (RPl_{p}). This proves that max (RPl_{p}) ≤ sup (Pl_{p}), which combined with our previous results gives d^{∗} = inf (Dl_{p}) = b^{T} ŷ = max (RPl_{p}) ≤ sup (Pl_{p}) = p^{∗}. Finally, using the weak duality of Theorem 4.5, i.e. p^{∗} ≤ d^{∗}, we obtain d^{∗} = inf (Dl_{p}) = b^{T} ŷ = sup (Pl_{p}) = p^{∗}, which implies that ŷ is optimum for (Pl_{p}), sup (Pl_{p}) = max (Pl_{p}) and finally the desired result p^{∗} = max (Pl_{p}) = inf (Dl_{p}) = d^{∗}.

4.3.3 Examples

We conclude this section by providing a few examples of the possible situations that can arise for a couple of primal-dual l_{p}-norm optimization problems. Let us consider the following problem data:

r = 1, K = {1}, n = 1, I_{1} = {1}, m = 1, A = 1, F = 0, c = 5, d ∈ ℝ, b = 1, p = 3

(d_{1} is left unspecified), which translates into the following primal problem:

sup y_{1} s.t. \frac{1}{3} |5 − y_{1}|^{3} ≤ d_{1} .  (Pl_{p})

Noting q = \frac{3}{2}, we can also write down the dual

inf 5x_{1} + d_{1}z_{1} + z_{1} \frac{1}{3/2} |\frac{x_{1}}{z_{1}} |^{3/2} s.t. x_{1} = 1, z_{1} ≥ 0, z_{1} = 0 ⇒ x_{1} = 0 .  (Dl_{p})

This pair of problems can readily be simplified to

sup y_{1} s.t. |5 − y_{1}| ≤ \sqrt[3]{3d_{1}} and inf 5 + d_{1}z_{1} + \frac{2}{3\sqrt{z_{1}}} s.t. z_{1} > 0

⋄ When d = 9, our primal constraint becomes |5 − y_{1}| ≤ 3, which gives a primal optimum equal to y_{1} = 8. Looking at the dual, we have

9z_{1} + \frac{2}{3\sqrt{z_{1}}} = \frac{1}{3}(27z_{1}) + \frac{2}{3}(\frac{1}{\sqrt{z_{1}}} ) ≤ (27z_{1})^{\frac{1}{3}} (\frac{1}{\sqrt{z_{1}}} )^{\frac{2}{3}} = 3

(using the weighted arithmetic-geometric mean), which shows that the dual optimum is also equal to 8, and is attained for (x, z) = (1, \frac{1}{9}). This is the most common situation: both optimum values are finite and attained, with a zero duality gap.

⋄ When d = 0, our primal constraint becomes |5 − y_{1}| ≤ 0, which implies that the only feasible solution is y_{1} = 5, giving a primal optimum equal to 5. The dual optimum value is then inf 5 + \frac{2}{3\sqrt{z_{1}}} = 5, equal to the primal but not attained (z_{1} → +∞). This shows that there are problems for which the dual optimum is not attained, i.e. we do not have the perfect duality of linear optimization (one can observe that in this case the primal had no strict interior).

⋄ Finally, when d = −1, the primal becomes infeasible while the dual is unbounded (take again z → +∞).

4.4 Complexity

The goal of this section is to prove it is possible to solve an l_{p}-norm optimization problem up to a given accuracy in polynomial time. According to the theoretical framework of Nesterov and Nemirovski [NN94], which was presented in Chapter 2, in order to solve the conic problem described in Chapter 3

inf_{x} c^{T} x s.t. Ax = b and x ∈ C ,  (CP)

we only need to find a computable self-concordant barrier function for the cone C, according to Definition 2.2. Indeed, we can apply for example the following variant of Theorem 2.5.

Theorem 4.8. Given a (κ, ν)-self-concordant barrier for the cone C ⊆ ℝ^{n} and a feasible interior starting point x_{0} ∈ int C satisfying δ(x_{0}, μ_{0}) < \frac{1}{13.42κ}, a short-step interior-point algorithm can solve problem (CP) up to ϵ accuracy within

O (κ\sqrt{θ} log \frac{μ_{0}κ\sqrt{ν}}{ϵ} ) iterations,

such that at each iteration the self-concordant barrier and its first and second derivatives have to be evaluated and a linear system has to be solved in ℝ^{n} (i.e. the Newton step for the barrier problem has to be computed).

We are now going to describe a self-concordant barrier that allows us to solve conic problems involving our L^{p} cone (we follow an approach similar to the one used in [XY00]). The following convex cone

{(x, y) ∈ ℝ × ℝ_{+} | |x|^{p} ≤ y}

(with p > 1) admits the well-known self-concordant barrier

f_{p} : ℝ × ℝ_{++} ↦ ℝ : (x, y) ↦ −2 log y − log(y^{2/p} − x^{2})

with parameters (1, 4) (see [NN94, Propostion 5.3.1], note we are using here the convention κ = 1). Let n ∈ ℕ, p ∈ ℝ^{n} and I = {1, 2, …, n}. We have that

{(x, y) ∈ ℝ^{n} × ℝ_{+}^{n} | |x_{i}|^{p_{i}} ≤ y_{i} ∀i ∈ I}

admits

f_{p} : ℝ^{n} × ℝ_{++}^{n} ↦ ℝ : (x, y) ↦ ∑_{i=1}^{n} (−2 log y_{i} − log(y_{i}^{2/p_{i}} − x_{i}^{2}))

with parameters (1, 4n) (using [NN94, Propostion 5.1.2]). This also implies that the set

S_{p} = \{(x, y, κ) ∈ ℝ^{n} × ℝ_{+}^{n} × ℝ | |x_{i}|^{p_{i}} ≤ y_{i} ∀i ∈ I and κ = ∑_{i=1}^{n} \frac{y_{i}}{p_{i}} \}

admits a self-concordant barrier f′_{p}(x, y, κ) = f_{p}(x, y) with parameters (1, 4n) (taking the cartesian product with ℝ essentially leaves the self-concordant barrier unchanged, taking the intersection with an affine subspace does not influence self-concordancy). Finally, we use another result from Nesterov and Nemirovski to find a self-concordant barrier for the conic hull of S_{p}, which is defined by

H_{p} = cl \{(x, t) ∈ S_{p} × ℝ_{++} | \frac{x}{t} ∈ S_{p}\}
= cl \{(x, y, κ, θ) ∈ S_{p} × ℝ_{++} | (\frac{x}{θ}, \frac{y}{θ} , \frac{κ}{θ}) ∈ S_{p}\}
= cl \{(x, y, κ, θ) ∈ ℝ^{n} × ℝ_{+}^{n} × ℝ × ℝ_{++} | |\frac{x_{i}}{θ} |^{p_{i}} ≤ \frac{y_{i}}{θ} ∀i ∈ I and \frac{κ}{θ} = ∑_{i=1}^{n} \frac{y_{i}}{p_{i}θ} \}

= cl \{(x, y, κ, θ) ∈ ℝ^{n} × ℝ_{+}^{n} × ℝ × ℝ_{++} | \frac{|x_{i}|^{p_{i}}}{θ^{p_{i}−1}} ≤ y_{i} ∀i ∈ I and κ = ∑_{i=1}^{n} \frac{y_{i}}{p_{i}} \}

= \{(x, y, κ, θ) ∈ ℝ^{n} × ℝ_{+}^{n} × ℝ × ℝ_{+} | \frac{|x_{i}|^{p_{i}}}{θ^{p_{i}−1}} ≤ y_{i} ∀i ∈ I and κ = ∑_{i=1}^{n} \frac{y_{i}}{p_{i}} \}

(to find the last equality, you have to consider accumulation points with θ = 0, which in fact must satisfy x = 0, which in turn can be seen to match exactly the convention about zero denominators we chose in Definition 4.1), and find that

h_{p} : ℝ^{n} × ℝ_{++}^{n} × ℝ × ℝ_{++} ↦ ℝ : (x, y, κ, θ) ↦ (f_{p}(\frac{x}{θ}, \frac{y}{θ}) − 8n log θ)

is a self-concordant barrier for H_{p} with parameter (20, 8n) (see [NN94, Proposition 5.1.4]). We now make the following interesting observation linking H_{p} to our cone L^{p}.

Theorem 4.9. The L^{p} cone is equal to the projection of H_{p} on the space of (x, κ, θ), i.e.

(x, θ, κ) ∈ L^{p} ⇔ ∃y ∈ ℝ_{+}^{n} | (x, y, κ, θ) ∈ H_{p} .

Proof. This proof is straightforward. First note that both sets take the same convention in case of a zero denominator. Let (x, θ, κ) ∈ L^{p}. Choosing y such that y_{i} = \frac{|x_{i}|^{p_{i}}}{θ^{p_{i}−1}} for all i ∈ I ensures that

∑_{i=1}^{n} \frac{y_{i}}{p_{i}} = ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} ≤ κ

(this last inequality because of the definition of L^{p}). It is now possible to increase y_{1} until the equality κ = ∑_{i=1}^{n} \frac{y_{i}}{p_{i}} is satisfied, which shows (x, y, κ, θ) ∈ H_{p}. For the reverse inclusion, suppose (x, y, κ, θ) ∈ H_{p}. This implies that

κ = ∑_{i=1}^{n} \frac{y_{i}}{p_{i}} ≥ ∑_{i=1}^{n} \frac{|x_{i}|^{p_{i}}}{p_{i}θ^{p_{i}−1}} ,

which is exactly the defining inequality of L^{p}.

Suppose now we have now to solve

inf_{x} c^{T} x s.t. Ax = b and x ∈ L^{p} .  (4.9)

In light of the previous theorem, it is equivalent to solve

inf_{(x,y)} c^{T} x s.t. Ax = b and (x, y) ∈ H_{p} ,

for which we know a self-concordant barrier with parameter (20, 8n). This implies that it is possible to find an approximate solution to problem (4.9) with accuracy ϵ in O(\sqrt{n} log \frac{1}{ϵ}) iterations. Moreover, since it is possible to compute in polynomial time the value of h_{p} and of its first two derivatives, we can conclude that problem (4.9) is solvable in polynomial time.

This argument is rather easy to generalize to the case of the cartesian product of several L^{p} cones or dual L_{s}^{q} cones, which shows eventually that any primal or dual l_{p}-norm optimization can be solved up to a given accuracy in polynomial time.

4.5 Concluding remarks

In this chapter, we have formulated l_{p}-norm optimization problems in a conic way and applied results from the standard conic duality theory to derive their special duality properties.

This leads in our opinion to clearer proofs, the specificity of the class of problems under study being confined to the convex cone used in the formulation. Moreover, the fundamental reason why this class of optimization problems has better duality properties than a general convex problem becomes clear: this is essentially due to the existence of a strictly interior dual solution (even if a reduction procedure involving an equivalent regularized problem has to be introduced when the original dual lacks a strictly feasible point).

It is also worthy to note that this is an example of nonsymmetric conic duality, i.e. involving cones that are not self-dual, unlike the very well-studied cases of linear, secondorder and semidefinite optimization.

Another advantage of this approach is the ease to prove polynomial complexity for our problems: finding a suitable self-concordant barrier is essentially all that is needed.

In the special case where all p_{i}’s are equal, one might think it is possible to derive those duality results with a simpler formulation relying on the standard cone involving p-norms, i.e. the p-cone defined as

𝕃_{p}^{n} = {(x, κ) ∈ ℝ^{n} × ℝ_{+} | ∥x∥_{p} ≤ κ} = {(x, κ) ∈ ℝ^{n} × ℝ_{+} | ∑_{i=1}^{n} |x_{i}|^{p} ≤ κ^{p}} .

However, we were note able to reach that goal, the reason being that the homogenizing variables θ and κ^{∗} appear to play a significant role in our approach and cannot be avoided.

Finally, we mention that this framework is general enough to be applied to other classes of structured convex problems. Chapter 5 will indeed deal with the class of problems known as geometric optimization.

Geometric optimization

Geometric optimization is an important class of problems that has many applications, especially in engineering design. In this chapter, we provide new simplified proofs for the well-known associated duality theory, using conic optimization. After introducing suitable convex cones and studying their properties, we model geometric optimization problems with a conic formulation, which allows us to apply the powerful duality theory of conic optimization and derive the duality results valid for geometric optimization.

5.1 Introduction

Geometric optimization forms an important class of problems that enables practitioners to model a large variety of real-world applications, mostly in the field of engineering design. We refer the reader to [DPZ67, Chapter V] for two detailed case studies in mechanical engineering (use of sea power) and electrical engineering (design of a transformer).

Although not convex itself, a geometric optimization problem can be easily transformed into a convex problem, for which a Lagrangean dual can be explicitly written. Several duality results are known for this pair of problems, some being mere consequences of convexity (e.g. weak duality), others being specific to this particular class of problems (e.g. the absence of a duality gap).

These properties were first studied in the sixties, and can be found for example in the reference book of Duffin, Peterson and Zener [DPZ67]. The aim of this chapter is to derive these results using the machinery of duality for conic optimization of Chapter 3, which has in our opinion the advantage of simplifying and clarifying the proofs.

In order to use this setting, we start by defining an appropriate convex cone that allows us to express geometric optimization problems as conic programs. The first step we take consists in studying some properties of this cone (e.g. closedness) and determine its dual. We are then in position to apply the general duality theory for conic optimization described in Chapter 3 to our problems and find in a rather seamless way the various well-known duality theorems of geometric optimization.

This chapter is organized as follows: we define and study in Section 5.2 the convex cones needed to model geometric optimization. Section 5.3 constitutes the main part of this chapter and presents new proofs of several duality theorems based on conic duality. Finally, we provide in Section 5.4 some hints on how to establish the link between our results and the classical theorems found in the literature, as well as some concluding remarks.

The approach we follow here is quite similar to the one we used in Chapter 4. However, geometric optimization differs from l_{p}-norm optimization in some important respects, which will be detailed later in this chapter.

5.2 Cones for geometric optimization

Let us introduce the geometric cone G^{n}, which will allow us to give a conic formulation of geometric optimization problems.

5.2.1 The geometric cone

Definition 5.1. Let n ∈ ℕ. The geometric cone G^{n} is defined by

G^{n} = \{(x, θ) ∈ ℝ_{+}^{n} × ℝ_{+} | ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ 1\}

using in the case of a zero denominator the following convention:

e^{− \frac{x_{i}}{0}} = 0 .

We observe that this convention results in (x, 0) ∈ G^{n} for all x ∈ ℝ_{+}^{n}. As special cases, we mention that G^{0} is the nonnegative real line ℝ_{+}, while G^{1} is easily shown to be equal to the 2-dimensional nonnegative orthant ℝ_{+}^{2}.

In order to use the powerful duality theory outlined in Chapter 3, we first have to prove that G^{n} is a convex cone.

Theorem 5.1. G^{n} is a convex cone.

Proof. To prove that a set is a convex cone, it suffices to show that it is closed under addition and nonnegative scalar multiplication (Definition 3.1 and Theorem 3.1). Indeed, if (x, θ) ∈ G^{n}, (x′, θ′) ∈ G^{n} and λ ≥ 0, we have

∑_{i=1}^{n} e^{− \frac{λx_{i}}{λθ}} = \begin{cases}∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ 1 & if λ > 0 \\ 0 ≤ 1 & if λ = 0 \end{cases}

which shows that λ(x, θ) ∈ G^{n}. Looking now at (x, θ) + (x′, θ′), we first consider the case θ > 0 and θ′ > 0 and write

∑_{i=1}^{n} e^{− \frac{x_{i}+x′_{i}}{θ+θ′}} = ∑_{i=1}^{n} (e^{− \frac{x_{i}}{θ}} )^{\frac{θ}{θ+θ′}} (e^{− \frac{x′_{i}}{θ′}} )^{\frac{θ′}{θ+θ′}} .

We can now apply Lemma 4.1 on each term of the sum, using vector (e^{− \frac{x_{i}}{θ}} , e^{− \frac{x′_{i}}{θ′}}) and weights (\frac{θ}{θ+θ′} , \frac{θ′}{θ+θ′}), satisfying \frac{θ}{θ+θ′} + \frac{θ′}{θ+θ′} = 1, to obtain

∑_{i=1}^{n} e^{− \frac{x_{i}+x′_{i}}{θ+θ′}} ≤ ∑_{i=1}^{n} \frac{θ}{θ + θ′} (e^{− \frac{x_{i}}{θ}}) + \frac{θ′}{θ + θ′} (e^{− \frac{x′_{i}}{θ′}})

= \frac{θ}{θ + θ′} ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} + \frac{θ′}{θ + θ′} ∑_{i=1}^{n} e^{− \frac{x′_{i}}{θ′}}

≤ \frac{θ}{θ + θ′} 1 + \frac{θ′}{θ + θ′} 1 = 1 ,

while in the case of θ′ = 0 we have

∑_{i=1}^{n} e^{− \frac{x_{i}+x′_{i}}{θ+θ′}} = ∑_{i=1}^{n} e^{− \frac{x_{i}+x′_{i}}{θ}} ≤ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ 1

(the case θ = 0 is similar). We have thus shown that (x + x′, θ + θ′) ∈ G^{n} in all cases, and therefore that G^{n} is a convex cone.

We now proceed to prove some properties of the geometric cone G^{n}.

Theorem 5.2. G^{n} is closed.

Proof. Let {(x^{k}, θ^{k})} a sequence of points in ℝ^{n+1} such that (x^{k}, θ^{k}) ∈ G^{n} for all k and lim_{k→∞}(x^{k}, θ^{k}) = (x^{∞}, θ^{∞}). In order to prove that G^{n} is closed, it suffices to show that (x^{∞}, θ^{∞}) ∈ G^{n}. Let us distinguish two cases:

⋄ θ^{∞} > 0. Using the easily proven fact that functions (x_{i}, θ) ↦ e^{− \frac{x_{i}}{θ}} are continuous on ℝ_{+} × ℝ_{++}, we have that

∑_{i=1}^{n} e^{− \frac{x^{∞}_{i}}{θ^{∞}}} = ∑_{i=1}^{n} lim_{k→∞} e^{− \frac{x^{k}_{i}}{θ^{k}}} = lim_{k→∞} ∑_{i=1}^{n} e^{− \frac{x^{k}_{i}}{θ^{k}}} ≤ 1 ,

which implies (x^{∞}, θ^{∞}) ∈ G^{n}.

⋄ θ^{∞} = 0. Since (x^{k}, θ^{k}) ∈ G^{n}, we have x^{k} ≥ 0 and thus x^{∞} ≥ 0, which implies that (x^{∞}, 0) ∈ G^{n}.

In both cases, (x^{∞}, θ^{∞}) is shown to belong to G^{n}, which proves the claim.

In order to use the strong duality theorem, we now proceed to identify the interior of the geometric cone.

Theorem 5.3. The interior of G^{n} is given by

int G^{n} = \{(x, θ) ∈ ℝ_{++}^{n} × ℝ_{++} | ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} < 1\} .

Proof. A point x belongs to the interior of a set S if and only if there exists an open ball centered at x entirely included in S. Let (x, θ) ∈ G^{n}. We first note that (x, 0) cannot belong to int G^{n}, because every open ball centered at (x, 0) contains a point with a negative θ component, which does not belong to the cone G^{n}. Suppose θ > 0 and the inequality in the definition of G^{n} is satisfied with equality, i.e.

∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} = 1 .

Every open ball centered at (x, θ) contains a point (x′, θ′) with x′ < x and θ′ > θ, which satisfies then

∑_{i=1}^{n} e^{− \frac{x′_{i}}{θ′}} > ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} = 1

and is thus outside of G^{n}, implying (x, θ) ∉ int G^{n}. We now show that all the remaining points that do not satisfy one of the two conditions mentioned above, i.e. the points with θ > 0 satisfying the strict inequality, belong to the interior of G^{n}. Let (x, θ) one of these points, and B(ϵ) the open ball centered at (x, θ) with radius ϵ. Restricting ϵ to sufficiently small values (i.e. choosing ϵ < θ), we have for all points (x′, θ′) ∈ B(ϵ)

x_{i} − ϵ ≤ x′_{i} ≤ x_{i} + ϵ and 0 < θ − ϵ ≤ θ′ ≤ θ + ϵ ,

which implies

\frac{x′_{i}}{θ′} ≥ \frac{x − ϵ}{θ + ϵ} and thus ∑_{i=1}^{n} e^{− \frac{x′_{i}}{θ′}} ≤ ∑_{i=1}^{n} e^{− \frac{x_{i}−ϵ}{θ+ϵ}} for all (x′, θ′) ∈ B(ϵ) .  (5.1)

Taking the limit of the last right-hand side when ϵ → 0, we find

lim_{ϵ→0} ∑_{i=1}^{n} e^{− \frac{x_{i}−ϵ}{θ+ϵ}} = ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} < 1

(because of the continuity of functions (x_{i}, θ) ↦ e^{− \frac{x_{i}}{θ}} on ℝ_{+}×ℝ_{++}). Therefore we can assume the existence of a value ϵ^{∗} such that

∑_{i=1}^{n} e^{− \frac{x_{i}−ϵ^{∗}}{θ+ϵ^{∗}}} < 1 ,

which because of (5.1) will imply that

∑_{i=1}^{n} e^{− \frac{x′_{i}}{θ′}} < 1

for all (x′, θ′) ∈ B(ϵ^{∗}). This inequality, combined with θ′ > 0, is sufficient to prove that the open ball B(ϵ^{∗}) is entirely included in G^{n}, hence that (x, θ) ∈ int G^{n}.

Theorem 5.4. G^{n} is solid and pointed.

Proof. The fact that 0 ∈ G^{n} ⊆ ℝ_{+}^{n+1} implies that G^{n} ∩ −G^{n} = {0}, i.e. G^{n} is pointed (Definition 3.2). To prove it is solid (Definition 3.3), we simply provide a point belonging to its interior, for example (e, \frac{1}{n}) (where e stands for the all-one vector). We have then

∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} = ne^{−n} < 1 ,

because e^{n} > n for all n ∈ ℕ, and therefore (e, \frac{1}{n}) ∈ int G^{n}.

To summarize, G^{n} is a solid pointed close convex cone, hence suitable for conic optimization.

5.2.2 The dual geometric cone

In order to express the dual of a conic problem involving the geometric cone G^{n}, we need to find an explicit description of its dual.

Theorem 5.5. The dual of G^{n} is given by

(G^{n})^{∗} = \{(x^{∗}, θ^{∗}) ∈ ℝ_{+}^{n} × ℝ | θ^{∗} ≥ ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log \frac{x_{i}^{∗}}{∑_{i=1}^{n} x_{i}^{∗}} \} .

Proof. Using Definition 3.4 for the dual cone, we have

(G^{n})^{∗} = {(x^{∗}, θ^{∗}) ∈ ℝ^{n} × ℝ | (x, θ)^{T} (x^{∗}, θ^{∗}) ≥ 0 for all (x, θ) ∈ G^{n}}

(the ^{∗} superscript on variables x^{∗} and θ^{∗} is a reminder of their dual nature). This condition on (x^{∗}, θ^{∗}) is equivalent to saying that the following infimum

δ(x^{∗}, θ^{∗}) = inf x^{T} x^{∗} + θθ^{∗} s.t. (x, θ) ∈ G^{n} .

has to be nonnegative. Let us distinguish the cases θ = 0 and θ > 0: we have that

δ(x^{∗}, θ^{∗}) = min{δ_{1}(x^{∗}, θ^{∗}), δ_{2}(x^{∗}, θ^{∗})}

with

\{ \begin{matrix}δ_{1}(x^{∗}, θ^{∗}) = inf x^{T} x^{∗} + θθ^{∗} s.t. (x, θ) ∈ G^{n} and θ = 0 \\ δ_{2}(x^{∗}, θ^{∗}) = inf x^{T} x^{∗} + θθ^{∗} s.t. (x, θ) ∈ G^{n} and θ > 0 \end{matrix} .

The first of these infima can be rewritten as

inf x^{T} x^{∗} s.t. x ≥ 0 ,

since (x, 0) ∈ G^{n} ⇔ x ≥ 0. It is easy to see that this infimum is equal to 0 if x^{∗} ≥ 0 and to −∞ when x^{∗} ≱ 0. Since we are looking for points with a nonnegative infimum δ(x^{∗}, θ^{∗}), we will require in the rest of this proof x^{∗} to be nonnegative and only consider the second infimum, which is equal to

inf θ [\frac{x^{T} x^{∗}}{θ} + θ^{∗}] s.t. ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ 1 and (x, θ) ∈ ℝ_{+}^{n} × ℝ_{++} .  (5.2)

Let us again distinguish two cases. When x^{∗} = 0, this infimum becomes

inf θθ^{∗} s.t. ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ 1 and (x, θ) ∈ ℝ_{+}^{n} × ℝ_{++} ,

which is nonnegative if and only if θ^{∗} ≥ 0, since θ can take any value in the open positive interval ]0 + ∞[. On the other hand, if x^{∗} ≠ 0, we have ∑_{i=1}^{n} x_{i}^{∗} > 0 and can define the auxiliary variables w_{i}^{∗} by

w_{i}^{∗} = \frac{x_{i}^{∗}}{∑_{i=1}^{n} x_{i}^{∗}}

(in order to simplify notations). We write the following chain of inequalities

1 ≥ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≥ ∑_{i|w_{i}^{∗}>0} e^{− \frac{x_{i}}{θ}} = ∑_{i|w_{i}^{∗}>0} w_{i}^{∗} (\frac{e^{− \frac{x_{i}}{θ}}}{w_{i}^{∗}} ) ≥ ∏_{i|w_{i}^{∗}>0} (\frac{e^{− \frac{x_{i}}{θ}}}{w_{i}^{∗}} )^{w_{i}^{∗}}  (5.3)

The second inequality comes from the fact that each term of the sum is positive (we remove some terms), and the third one uses Lemma 4.1 with weights w_{i}^{∗}, noting that ∑_{i|w_{i}^{∗}>0} w_{i}^{∗} = ∑_{i=1}^{n} w_{i}^{∗} = 1. From this last inequality we derive successively

∏_{i|w_{i}^{∗}>0} e^{− \frac{x_{i}w^{∗}_{i}}{θ}} ≤ ∏_{i|w_{i}^{∗}>0} w_{i}^{∗}^{w_{i}^{∗}} ,

− ∑_{i|w_{i}^{∗}>0} \frac{x_{i}w_{i}^{∗}}{θ} ≤ ∑_{i|w_{i}^{∗}>0} w_{i}^{∗} log w_{i}^{∗} (taking the logarithms) ,

∑_{i=1}^{n} \frac{x_{i}x_{i}^{∗}}{θ} ≥ − ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} (multiplying by − ∑_{i=1}^{n} x_{i}^{∗}) ,

\frac{x^{T} x^{∗}}{θ} + θ^{∗} ≥ θ^{∗} − ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} , and finally

inf_{(x,θ)∈G^{n}|θ>0} \frac{x^{T} x^{∗}}{θ} + θ^{∗} ≥ θ^{∗} − ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} .

Examining carefully the chain of inequalities in (5.3), we observe that a suitable choice of (x, θ) can lead to attainment of this last infimum: namely, we need to have ⋄ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} = 1, for the first inequality in (5.3),

⋄ x_{i} → +∞ for all indices i such that w_{i}^{∗} = 0, in order to have e^{− \frac{x_{i}}{θ}} → 0 when w_{i}^{∗} = 0 for the second inequality in (5.3),

⋄ all terms (\frac{e^{− \frac{x_{i}}{θ}}}{w_{i}^{∗}} ) with indices such that w_{i}^{∗} > 0 equal to each other, for the third inequality in (5.3).

These conditions are compatible: summing up the constant terms, we find

\frac{e^{− \frac{x_{i}}{θ}}}{w_{i}^{∗}} (when w_{i}^{∗} > 0) = \frac{∑_{i|w_{i}^{∗}>0} e^{− \frac{x_{i}}{θ}}}{∑_{i|w_{i}^{∗}>0} w_{i}^{∗}} = ∑_{i|w_{i}^{∗}>0} e^{− \frac{x_{i}}{θ}} → ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} = 1 ,

which gives e^{− \frac{x_{i}}{θ}} = w_{i}^{∗} for all i such that w_{i}^{∗} > 0. Summarizing, we can choose x according to

\begin{cases}x_{i} = −θ log w_{i}^{∗} & when w_{i}^{∗} > 0 \\ x_{i} → +∞ & when w_{i}^{∗} = 0 \end{cases} ,

which proves that

inf_{(x,θ)∈G^{n}|θ>0} \frac{x^{T} x^{∗}}{θ} + θ^{∗} = θ^{∗} − ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} .  (5.4)

Since the additional multiplicative θ in (5.2) doesn’t change the sign of this infimum (because θ > 0), we may conclude that it is nonnegative if and only if

θ^{∗} − ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} ≥ 0 .

Combining with the special case x^{∗} = 0 and the constraint x^{∗} ≥ 0 implied by the first infimum, we conclude that the dual cone is given by

(G^{n})^{∗} = {(x^{∗}, θ^{∗}) ∈ ℝ_{+}^{n} × ℝ | θ^{∗} ≥ ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗}} ,

as announced.

As special cases, since G^{0} = ℝ_{+} and G^{1} = ℝ_{+}^{2}, we may check that (G^{0})^{∗} = (ℝ_{+})^{∗} = ℝ_{+} and (G^{1})^{∗} = (ℝ_{+}^{2})^{∗} = ℝ_{+}^{2}, as expected. These two cones are thus self-dual, but it is easy to see that geometric cones of higher dimension are not self-dual any more. To illustrate our purpose, we provide in Figure 5.1 the three-dimensional graphs of the boundary surfaces of G^{2} and (G^{2})^{∗}.

Note 5.1. Since we have 0 ≤ w_{i}^{∗} ≤ 1 for all indices i, each logarithmic term appearing in this definition is nonpositive, as well as their sum, which means that (x^{∗}, θ^{∗}) ∈ (G^{n})^{∗} as soon as x^{∗} and θ^{∗} are nonnegative. This fact could have been guessed prior to any computation: noticing that G^{n} ⊆ ℝ_{+}^{n+1} and (ℝ_{+}^{n+1})^{∗} = ℝ_{+}^{n+1}, we immediately have that (G^{n})^{∗} ⊇ ℝ_{+}^{n+1}, because taking the dual of a set inclusion reverses its direction.

Figure 5.1: The boundary surfaces of G^{2} and (G^{2})^{∗}.

Finding the dual of G^{n} was a little involved, but establishing its properties is straightforward.

Theorem 5.6. (G^{n})^{∗} is a solid, pointed, closed convex cone. Moreover, ((G^{n})^{∗})^{∗} = G^{n}.

Proof. The proof of this fact is immediate by Theorem 3.3 since (G^{n})^{∗} is the dual of a solid, pointed, closed convex cone.

The interior of (G^{n})^{∗} is also rather easy to obtain:

Theorem 5.7. The interior of (G^{n})^{∗} is given by

int(G^{n})^{∗} = \{(x^{∗}, θ^{∗}) ∈ ℝ_{++}^{n} × ℝ | θ^{∗} > ∑_{i=1}^{n} x_{i}^{∗} log \frac{x_{i}^{∗}}{∑_{i=1}^{n} x_{i}^{∗}} \} .

Proof. We first note that (G^{n})^{∗}, a convex set, is the epigraph of the following function

f_{n} : ℝ_{+}^{n} ↦ ℝ : x ↦ ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log \frac{x_{i}^{∗}}{∑_{i=1}^{n} x_{i}^{∗}} ,

which implies that f_{n} is convex (by definition of a convex function). Hence we can apply Lemma 7.3 in [Roc70a] to get

int(G^{n})^{∗} = int epi f_{n} = {(x^{∗}, θ^{∗}) ∈ int dom f_{n} × ℝ | θ^{∗} > f_{n}(x^{∗})} ,

which is exactly our claim since int ℝ_{+}^{n} = ℝ_{++}^{n}.

The last piece of information we need about the pair of cones (G^{n}, (G^{n})^{∗}) is its set of orthogonality conditions.

Theorem 5.8 (orthogonality conditions). Let v = (x, θ) ∈ G^{n} and v^{∗} = (x^{∗}, θ^{∗}) ∈ (G^{n})^{∗}. We have v^{T} v^{∗} = 0 if and only if one of these two sets of conditions is satisfied

θ = 0 and x_{i}x_{i}^{∗} = 0 for all i
θ > 0 and \{ \begin{matrix}∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} & = & θ^{∗} \\ (∑_{i=1}^{n} x_{i}^{∗})e^{− \frac{x_{i}}{θ}} & = & x_{i}^{∗} for all i \end{matrix} .

Proof. To prove this fact, we merely have to reread carefully the proof of Theorem 5.5, paying attention to the cases where the infimum is equal to zero. In the first case examined, θ = 0, we have v^{T} v^{∗} = x^{T} x^{∗}. Since x and x^{∗} are two nonnegative vectors, we have v^{T} v^{∗} = 0 if and only if x_{i}x_{i}^{∗} = 0 for every index i, which gives the first set of conditions of the theorem.

When θ > 0, we first have the special case x^{∗} = 0 which gives v^{T} v^{∗} = θθ^{∗}. This quantity can only be zero if θ^{∗} = 0, i.e. when (x^{∗}, θ^{∗}) = 0. When x^{∗} ≠ 0, the proof of Theorem 5.5 shows that v^{T} v^{∗} can only be zero when the infimum (5.4) is equal to zero and attained, which implies θ^{∗} = ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗}. However, this infimum is not always attained by a finite vector (x, θ), because of the condition x_{i} → +∞ that is required when w_{i}^{∗} = 0. The scalar product v^{T} v^{∗} is thus equal to zero only if all w_{i}^{∗}’s are positive, i.e. when all x_{i}^{∗}’s are positive: in this case, the two sets of equalities θ^{∗} = ∑_{i|x_{i}^{∗}>0} x_{i}^{∗} log w_{i}^{∗} (to have a zero infimum) and e^{− \frac{x_{i}}{θ}} = w_{i}^{∗} (to attain the infimum) must be satisfied.

Rephrasing this last equality as (∑_{i=1}^{n} x_{i}^{∗})e^{− \frac{x_{i}}{θ}} = x_{i}^{∗} to take into account the special case (x^{∗}, θ^{∗}) = 0, we find the second set of conditions of our theorem.

5.3 Duality for geometric optimization

In this section, we introduce a form of geometric optimization problems that is suitable to our purpose and prove several duality properties using the previously defined primal-dual pair of convex cones. These results are well-known and can be found e.g. in [DPZ67]. However, our presentation differs and handles problems expressed in a slightly different (but equivalent) format, and hence provides results adapted to the formulation we use. We refer the reader to Subsection 5.4.1 where the connection is made between our results and their classical counterparts.

5.3.1 Conic formulation

We start with the original formulation of a geometric optimization problem (see e.g. [DPZ67]). Let us define two sets K = {0, 1, 2, …, r} and I = {1, 2, …, n} and let {I_{k}}_{k∈K} be a partition of I into r + 1 classes, i.e. satisfying

∪_{k∈K}I_{k} = I and I_{k} ∩ I_{l} = ∅ for all k ≠ l .

The primal geometric optimization problem is the following:

inf G_{0}(t) s.t. t ∈ ℝ_{++}^{m} and G_{k}(t) ≤ 1 for all k ∈ K \ {0} ,  (OGP)

where t is the m-dimensional column vector we want to optimize and the functions G_{k} defining the objective and the constraints are so-called posynomials, given by

G_{k} : ℝ_{++}^{m} ↦ ℝ_{++} : t ↦ ∑_{i∈I_{k}} C_{i} ∏_{j=1}^{m} t_{j}^{a_{ij}} ,

where exponents a_{ij} are arbitrary real numbers and coefficients C_{i} are required to be strictly positive (hence the name posynomial). These functions are very well suited for the formulation of constraints that come from the laws of physics or economics (either directly or using an empirical fit).

Although not convex itself (choose for example G_{0} : t ↦ t^{1/2} as the objective, which is not a convex function), a geometric optimization problem can be easily transformed into a convex problem, for which a Lagrangean dual can be explicitly written. This transformation uses the following change of variables:

t_{j} = e^{y_{j}} for all j ∈ {1, 2, …, m} ,  (5.5)

to become

inf g_{0}(y) s.t. g_{k}(y) ≤ 1 for all k ∈ K \ {0} .  (OGP’)

The functions g_{k} are defined to satisfy g_{k}(y) = G_{k}(t) when (5.5) holds, which means

g_{k} : ℝ^{m} ↦ ℝ_{++} : y ↦ ∑_{i∈I_{k}} C_{i} ∏_{j=1}^{m} (e^{y_{j}})^{a_{ij}} = ∑_{i∈I_{k}} e^{−c_{i}+∑_{j=1}^{m} y_{j}a_{ij}} = ∑_{i∈I_{k}} e^{a_{i}^{T} y−c_{i}} ,

where the coefficient vector c ∈ ℝ^{n} is given by c_{i} = − log C_{i} and a_{i} = (a_{i1}, a_{i2}, …, a_{im})^{T} is an m-dimensional column vector. Note that unlike the original variables t and coefficients C, variables y and coefficients c are not required to be strictly positive and can take any real value.

It is straightforward to check that functions g_{k} are now convex, hence that (OGP’) is a convex optimization problem. However, we will not establish convexity directly but rather derive it from the fact that problem (OGP’) can be cast as a conic optimization problem. Moreover, following others [Kla74, dJRT95, RT98], we will not use this formulation but instead work with a slight variation featuring a linear objective:

sup b^{T} y s.t. g_{k}(y) ≤ 1 for all k ∈ K ,  (GP)

where b ∈ ℝ^{m} and 0 has been removed from set K.

It will be shown later that problems in the form (OGP’) (and (OGP)) can be expressed in this format, and the results we are going to obtain about problem (GP) will be translated back to these more traditional settings later in Subsection 5.4.1. We can focus our attention on formulation (GP) without any loss of generality.

Let us now model problem (GP) with a conic formulation. As in Chapter 4, we will use the following useful convention: v_{S} (resp. M_{S}) denotes the restriction of column vector v (resp. matrix M) to the components (resp. rows) whose indices belong to set S. We introduce a vector of auxiliary variables s ∈ ℝ^{n} to represent the exponents used in functions g_{k}, more precisely we let

s_{i} = c_{i} − a_{i}^{T} y for all i ∈ I or, in matrix form, s = c − A^{T} y ,

where A is a m × n matrix whose columns are a_{i}. Our problem becomes then

sup b^{T} y s.t. s = c − A^{T} y and ∑_{i∈I_{k}} e^{−s_{i}} ≤ 1 for all k ∈ K ,

which is readily seen to be equivalent to the following, using the definition of G^{n} (where variables θ have been fixed to 1),

sup b^{T} y s.t. A^{T} y + s = c and (s_{I_{k}}, 1) ∈ G^{#I_{k}} for all k ∈ K ,

and finally to

sup b^{T} y s.t. (\begin{matrix}A^{T} \\ 0 \end{matrix} ) y + (\begin{matrix}s \\ v \end{matrix}) = (\begin{matrix}c \\ e \end{matrix}) and (s_{I_{k}}, v_{k}) ∈ G^{n_{k}} for all k ∈ K ,  (CGP)

where e is the all-one vector in ℝ^{r}, n_{k} = #I_{k} and an additional vector of fictitious variables v ∈ ℝ^{r} has been introduced, whose components are fixed to 1 by part of the linear constraints. This is exactly a conic optimization problem, in the dual form (CD), using variables (ỹ, s̃), data (Ã, b̃, c̃) and a cone K^{∗} such that

ỹ = y, s̃ = (\begin{matrix}s \\ v \end{matrix}) , Ã = (A 0) , b̃ = b, c̃ = (\begin{matrix}c \\ e \end{matrix}) and K^{∗} = G^{n_{1}} × G^{n_{2}} × ⋯ × G^{n_{r}} ,

where K^{∗} has been defined according to Note 3.1, since we have to deal with multiple conic constraints involving disjoint sets of variables.

Using properties of G^{n} and (G^{n})^{∗} proved in the previous section, it is straightforward to show that K^{∗} is a solid, pointed, closed convex cone whose dual is

(K^{∗})^{∗} = K = (G^{n_{1}})^{∗} × (G^{n_{2}})^{∗} × ⋯ × (G^{n_{r}})^{∗} ,

another solid, pointed, closed convex cone, according to Theorem 5.6. This allows us to derive a dual problem to (CGP) in a completely mechanical way and find the following conic optimization problem, expressed in the primal form (CP):

inf (c^{T} e^{T} ) (\begin{matrix}x \\ z \end{matrix}) s.t. (A 0) (\begin{matrix}x \\ z \end{matrix}) = b and (x_{I_{k}}, z_{k}) ∈ (G^{n_{k}})^{∗} for all k ∈ K , (CGD)

where x ∈ ℝ^{n} and z ∈ ℝ^{r} are the vectors we optimize. This problem can be simplified: making the conic constraints explicit, we find

inf c^{T} x + e^{T} z s.t. Ax = b, x_{I_{k}} ≥ 0 and z_{k} ≥ ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} for all k ∈ K ,

which can be further reduced to

inf c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} s.t. Ax = b and x ≥ 0 .  (GD)

Indeed, since each variable z_{k} is free except for the inequality coming from the associated conic constraint, these inequalities must be satisfied with equality at each optimum solution and variables z can therefore be removed from the formulation. As could be expected, the dual problem we have just found using conic duality and our primal-dual pair of cones (G^{n}, (G^{n})^{∗}) corresponds to the usual dual for problem (GP) found in the literature [Kla76, dJRT95]. We will also show later in Subsection 5.4.1 that it also allows us to derive the dual problem in the traditional formulations (OGP) and (OGP’). We end this section by pointing out that, up to now, our reasoning has been completely similar to the one used for l_{p}-norm optimization in Chapter 4.

5.3.2 Duality theory

We are now about to apply the various duality theorems described in Chapter 3 to geometric optimization. Our strategy will be the following: in order to prove results about the pair (GP)–(GD), we are going to apply our theorems to the conic primal-dual pair (CGP)–(CGD) and use the equivalence that holds between (CGP) and (GP) and between (CGD) and (GD). We start with the weak duality theorem.

Theorem 5.9 (Weak duality). Let y a feasible solution for primal problem (GP) and x a feasible solution for dual problem (GD). We have

b^{T} y ≤ c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} ,  (5.6)

equality occurring if and only if

(∑_{i∈I_{k}} x_{i})e^{a_{i}^{T} y−c_{i}} = x_{i} for all i ∈ I_{k}, k ∈ K .

Proof (the original proof can be found in [Roc70b] or [Kla74, §1]). On the one hand, we note that y can be easily converted to a feasible solution (y, s, v) for the conic problem (CGP), simply by choosing vectors s and v according to the linear constraints. On the other hand, x can also be converted to a feasible solution (x, z) for the conic problem (CGD), admitting the same objective value, by choosing

z_{k} = ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} for all k ∈ K .  (5.7)

Applying now the weak duality Theorem 3.4 to the conic primal-dual pair (CGP)–(CGD) with feasible solutions (x, z) and (y, s, v), we find the announced inequality

b^{T} y ≤ c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} ,

equality occurring if and only if the orthogonality conditions given in Theorem 5.8 are satisfied for each conic constraint. Since θ corresponds here to v_{k}, which is always equal to 1 because of the linear constraints, we can rule out the first set of equalities (occurring where θ = 0) and keep only the second set of conditions. The first of these equalities being always satisfied because of our choice of z_{k}, we finally conclude that equality (5.6) can occur if and only if the following set of remaining equalities is satisfied, namely

(∑_{i∈I_{k}} x_{i})e^{− \frac{s_{i}}{v_{i}}} = x_{i} for all i ∈ I_{k}, k ∈ K ,

which is equivalent to our claim because of the linear constraints on s_{i} and v_{i}.

The following theorem is an application of the strong duality Theorem 3.5, and requires therefore the existence of a specific primal feasible solution.

Theorem 5.10. If there exists a feasible solution for the primal problem (GP) satisfying strictly the inequality constraints, i.e. a vector y such that

g_{k}(y) < 1 for all k ∈ K

we have either

⋄ an infeasible dual problem (GD) if primal problem (GP) is unbounded

⋄ a feasible dual problem (GD) whose optimum objective value is attained by a feasible vector x if primal problem (GP) is bounded. Moreover, the optimum objective values of (GP) and (GD) are equal.

Proof (a more classical proof can be found in [Kla74, §2]). Choosing again vectors s and v according to the linear constraints, we find a feasible solution (y, s, v) for the primal conic problem (CGP). Moreover, recalling the description of int G^{n} given by Theorem 5.3, the conditions v_{k} = 1 > 0 and g_{k}(y) = ∑_{i∈I_{k}} e^{−s_{i}} < 1 ensure that (y, s, v) is a strictly feasible solution for (CGP). The strong duality Theorem 3.5 implies then that we have either

⋄ an infeasible dual problem (CGD) if primal problem (CGP) is unbounded: this is equivalent to the first part of our claim, since it is clear that (CGP) is unbounded if and only if (GP) is unbounded and that (CGD) is infeasible if and only if (GD) is infeasible (indeed, (x, z) feasible for (CGD) implies x feasible for (GD), while x feasible for (GD) implies (x, 0) feasible for (CGD)). This fact could also have been obtained as a simple consequence of weak duality Theorem 5.9.

⋄ a feasible dual problem (CGD) whose optimum objective value is attained by a feasible vector (x, z) if primal problem (CGP) is bounded. Moreover, the optimum objective values of (CGP) and (CGD) are equal. Obviously, the finite optimum objective values of (CGP) and (GP) are equal. It is also clear that optimal variables z_{k} in (CGD) must attain the lower bounds defined by the conic constraints, as in (5.7), which implies that vector x is optimum for problem (GD) and has the same objective value as (x, z) in (CGD). This proves the second part of our claim.

Let us note again that a sufficient condition for the second case of this theorem to happen is the existence of a feasible solution for the dual problem (GD), because of the weak duality property.

The strong duality theorem can also be applied on the dual side.

Theorem 5.11. If there exists is a strictly positive feasible solution for the dual problem (GD), i.e. a vector x such that

Ax = b and x > 0 ,

we have either

⋄ an infeasible primal problem (GP) if dual problem (GD) is unbounded

⋄ a feasible primal problem (GP) whose optimum objective value is attained by a feasible vector y if dual problem (GD) is bounded. Moreover, the optimum objective values of (GD) and (GP) are equal.

Proof (a traditional proof can be found in [Kla74, §5]). As for the previous theorem, the first part of our claim is a direct consequence of Theorem 5.9, that does not really rely on the existence of a strictly positive x. Let us prove the second part of our claim and suppose that problem (GD) is bounded. Problem (CGD) cannot be unbounded, because each feasible solution (x, z) for (CGD) leads to a feasible x for (GD) with a lower objective (because of the conic constraints), which would also lead to an unbounded (GD). Using the description of int(G^{n})^{∗} given by Theorem 5.7, we find that a feasible x > 0 for (GD) can be easily converted to a strictly feasible solution (x, z) for (CGD), taking sufficiently large values for variables z_{k} (letting z_{k} = 1 for example is enough). The strong duality theorem implies thus, since (CGD) has been shown to be bounded, that problem (CGP) is feasible with an optimum objective value attained by a feasible vector (y, s, v) and equal to the dual optimum objective value of (CGD). Obviously, on the one hand, vector y is a feasible optimum solution to problem (GP), attaining the same objective value as (y, s, v) in (CGD). On the other hand, the finite optimum objective values of (CGD) and (GD) must be equal, even if no feasible solution is actually optimum (since x feasible for (GD) implies (x, z) feasible for (CGD) with the same objective value and (x, z) feasible for (CGD) implies x feasible for (GD) with a smaller or equal objective value). This is enough to prove the second part of our claim.

To conclude this section, we prove a last theorem that involves the alternate version of the strong duality theorem. Let us introduce the following family of optimization problems, parameterized by a strictly positive parameter δ:

p̂(δ) = sup b^{T} y s.t. g_{k}(y) ≤ e^{δ} for all k ∈ K .  (GP_{δ})

It is clear that each of these problems is a (strict) relaxation of problem (GP), because e^{δ} > 1 for δ > 0, hence we have p̂(δ) ≥ p^{∗} for all δ. Moreover, since the feasible region of these problems shrinks as δ tends to zero, p̂(δ) is a nondecreasing function of δ and we can always define the following limit

p̂ = lim_{δ→0^{+}} p̂(δ) ,

which we will call the subvalue of problem (GP). We have the following theorem

Theorem 5.12. If there exists a feasible solution to the dual problem (GD), the subvalue of the primal problem (GP) is equal to the optimum objective value of the dual problem (GD).

Proof. We are going to show in fact that the primal subvalue p̂ is equal to the subvalue p^{−} of the primal conic optimization problem (CGP) according to Definition 3.7. Using Theorem 3.6 on the primal-dual conic pair (CGP)–(CGD), we will find that p^{−} = d^{∗} (the first case of the theorem cannot happen since (GD), and hence (CGD), is feasible by hypothesis). Noting finally that the optimum objective values of (CGD) and (GD) are equal (which has been shown in the course of the previous proof) will conclude our proof.

Let us restate the definition of the subvalue p^{−} for problem (CGP). Defining the following family of problems, parameterized by a strictly positive parameter ϵ,

sup_{(y,s,v)} b^{T} y s.t. ∥(\begin{matrix}A^{T} \\ 0 \end{matrix} ) y + (\begin{matrix}s \\ v \end{matrix}) − (\begin{matrix}c \\ e \end{matrix})∥ < ϵ, (s_{I_{k}}, v_{k}) ∈ G^{n_{k}} ∀k ∈ K ,  (CGP_{ϵ})

whose optimum objective values will be denoted by p̄(ϵ), we have that the subvalue p^{−} of the primal problem (CGP) is defined by

p^{−} = lim_{ϵ→0^{+}} p̄(ϵ) .

We first show that for all δ > 0, the inequality p̂(δ) ≤ p̄(ϵ) holds for some well chosen value of ϵ. Let y a feasible solution for problem (GP_{δ}). Using the definition of g_{k}, constraints g_{k}(y) ≤ e^{δ} easily give

∑_{i∈I_{k}} e^{a_{i}^{T} y−c_{i}−δ} ≤ 1 ,

which shows that the following choice of vectors s and v

s_{i} = c_{i} − a_{i}^{T} y + δ for all i ∈ I and v_{k} = 1 for all k ∈ K

will be feasible for problem (CGP_{ϵ}) with ϵ = δ\sqrt{n}, since we have then (s_{I_{k}}, v_{k}) ∈ G^{n_{k}}∀k ∈ K and

∥(\begin{matrix}A^{T} \\ 0 \end{matrix} ) y + (\begin{matrix}s \\ v \end{matrix}) − (\begin{matrix}c \\ e \end{matrix})∥ = ∥(\begin{matrix}δ \\ 0 \end{matrix})∥ = δ\sqrt{n} .

Since every feasible solution y for (GP_{δ}) gives a feasible solution (y, s, v) for (CGP_{ϵ}) with the same objective value, the latter problem cannot have a smaller optimum objective value and we have p̂(δ) ≤ p̄(δ\sqrt{n}). Taking the limit when δ → 0, this shows that p̂ ≤ p^{−}.

Let us now work in the opposite direction and let (y, s, v) a feasible solution to problem (CGP_{ϵ}). We have thus

∑_{i∈I_{k}} e^{− \frac{s_{i}}{v_{k}}} ≤ 1 for all k ∈ K and ∥(\begin{matrix}A^{T} y + s − c \\ v − e \end{matrix})∥ < ϵ ,

which implies

\{ \begin{matrix}|a_{i}^{T} y + s_{i} − c_{i}| < ϵ for all i ∈ I \\ |v_{k} − 1| < ϵ for all k ∈ K \end{matrix} .

We write

1 ≥ ∑_{i∈I_{k}} e^{− \frac{s_{i}}{v_{i}}} > ∑_{i∈I_{k}} e^{− \frac{c_{i}−a^{T}_{i} y+ϵ}{1−ϵ}} ,

since v_{k} > 1 − ϵ, s_{i} < c_{i} − a_{i}^{T} y + ϵ and x ↦ e^{−x} is a monotonic decreasing function. Defining ỹ = \frac{y}{1−ϵ}, we have

\frac{c_{i} − a_{i}^{T} y + ϵ}{1 − ϵ} = c_{i} − a_{i}^{T} ỹ + \frac{c_{i} + ϵ}{1 − ϵ} − c_{i}
= c_{i} − a_{i}^{T} ỹ + \frac{ϵ}{1 − ϵ} (c_{i} + 1)
≤ c_{i} − a_{i}^{T} ỹ + \frac{ϵ}{1 − ϵ} (max c_{i} + 1)
≤ c_{i} − a_{i}^{T} ỹ + \frac{Cϵ}{1 − ϵ} ,

where C = max c_{i} + 1. We have thus

1 > ∑_{i∈I_{k}} e^{− \frac{c_{i}−a^{T}_{i} y+ϵ}{1−ϵ}} ≥ ∑_{i∈I_{k}} e^{a_{i}^{T} ỹ−c_{i}− \frac{Cϵ}{1−ϵ}} = e^{− \frac{Cϵ}{1−ϵ}} ∑_{i∈I_{k}} e^{a_{i}^{T} ỹ−c_{i}} ,

which shows that

∑_{i∈I_{k}} e^{a_{i}^{T} ỹ−c_{i}} < e^{\frac{Cϵ}{1−ϵ}} ,

i.e. ỹ is a feasible solution to problem (GP_{δ}) with δ = \frac{Cϵ}{1−ϵ}. Since this solution has an objective value b^{T} ỹ equal to b^{T} y divided by 1−ϵ, this means that p̄(ϵ) ≤ (1−ϵ)p̂(\frac{Cϵ}{1−ϵ}). Taking the limit when ϵ → 0, this shows that p^{−} ≤ p̂, and we may conclude that p^{−} = p̂, as announced.

5.3.3 Refined duality

The properties we have proved so far about our pair of primal-dual geometric optimization problems (GP)–(GD) are merely more or less direct consequences of their convex nature, hence valid for all convex optimization problems. In this section, we are going further and prove a result that does not hold in the general convex case, namely we show that our pair of primal-dual problems cannot have a strictly positive duality gap.

Theorem 5.13. If both problems (GP) and (GD) are feasible, their optimum objective values are equal (but not necessarily attained).

Proof (the original proof can be found in [Kla74, §7]). In Theorem 5.11, we proved the existence of a zero duality gap using some assumption on the dual, namely the existence of a strictly positive feasible vector. What we are going to show here is that if such a point does not exist, i.e. one or more components of vector x are zero for all feasible dual solutions, our primal-dual pair can be reduced to an equivalent pair of problems where these components have been removed, in other words a primal-dual pair with a strictly positive feasible dual solution and a zero duality gap.

In order to use this strategy, we start by identifying the components of x that are identically equal to zero on the dual feasible region. This can be done with the following linear optimization problem:

min 0 s.t. Ax = b and x ≥ 0 .  (BLP)

Since this problem has a zero objective function, all feasible solutions are optimal and we deduce that if a variable x_{i} is zero for all feasible solutions to problem (GD), it is zero for all optimal solution to problem (BLP). We are going to need the Goldman-Tucker Theorem 4.6 previously used in Chapter 4.

Writing the dual of problem (BLP)

max b^{T} y s.t. A^{T} y + s = 0 and s ≥ 0 ,  (BLD)

we find that both (BLP) and (BLD) are feasible (the former because (GD) is feasible, the latter because (y, s) = (0, 0) is always a feasible solution), and thus that the Goldman-Tucker theorem is applicable.

Having now the optimal partition (B, N) at hand, we observe that the index set N defines exactly the set of variables x_{i} that are identically zero on the feasible region of problem (GD). We are now able to introduce a reduced primal-dual pair of geometric optimization problems, where variables x_{i} with i ∈ N have been removed. We start with the dual problem

inf c_{B}^{T}x_{B} + ∑_{k∈K} ∑_{i∈I_{k}∩B|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}∩B} x_{i}} s.t. A_{B}x_{B} = b and x_{B} ≥ 0 .  (RGD)

It is straightforward to check that this problem is completely equivalent to problem (GD), since the variables we removed had no contribution to the objective or to the constraints in (GD). Indeed, there is a one-to-one correspondence preserving objective values between feasible solutions x_{B} for (RGD) and feasible solutions x for (GD), the latter satisfying always x_{N} = 0. Our primal geometric optimization problem becomes

sup b^{T} y s.t. g_{k}^{B}(y) ≤ 1 for all k ∈ K ,  (RGP)

where functions g_{k}^{B} are now defined over the sets I_{k} ∩ B, i.e.

g_{k}^{B} : ℝ^{m} ↦ ℝ_{++} : y ↦ ∑_{i∈I_{k}∩B} e^{a_{i}^{T} y−c_{i}} .

Since the Goldman-Tucker theorem implies the existence of a feasible vector x^{∗} such that x_{B}^{∗} > 0 and x_{N}^{∗} = 0, we find that x_{B}^{∗} is a strictly positive feasible solution to (RGD), which allows us to apply Theorem 5.11. Knowing that (GP) is feasible, problems (GD) and (RGD) must be bounded: we are in the second case of the theorem and can conclude that problem (RGP) attains an optimum objective value equal to the optimum objective value of problem (RGD). The last thing we have to show in order to finish our proof is that the optimum values of primal problem (GP) and its reduced version (RGP) are equal.

Let us start with ȳ, one of the optimal solutions to (RGP) that are known to exist. Our goal is thus to prove that problem (GP) has an optimum objective value equal to b^{T} ȳ. Unfortunately, ȳ is not always feasible for problem (GP), since the additional terms in g_{k} corresponding to indices i ∈ N result in g_{k}(ȳ) > g_{k}^{B}(ȳ) and possibly g_{k}(ȳ) > 1.

To solve this problem, we are going to perturb ȳ with a suitably chosen vector, in order to make it feasible. The existence of this perturbation vector will be again derived from the Goldman-Tucker theorem, in the following manner. Let (x^{∗}, y^{∗}, s^{∗}) a strictly complementary pair for problems (BLP)–(BLD). Since the optimum primal objective value is obviously equal to zero, we also have that the optimum dual objective b^{T} y^{∗} is equal to zero. Moreover, we have that A^{T} y^{∗} + s^{∗} = 0, which gives

A_{B}^{T}y^{∗} = −s_{B}^{∗} = 0 and A_{N}^{T} y^{∗} = −s_{N}^{∗} < 0 .

Considering a vector y defined by y = ȳ + λy^{∗}, where λ is a positive parameter that is going to tend to +∞, it is easy to check that

g_{k}(y) = g_{k}^{B}(y) + g_{k}^{N} (y)
= ∑_{i∈I_{k}∩B} e^{a_{i}^{T} y−c_{i}} + ∑_{i∈I_{k}∩N} e^{a_{i}^{T} y−c_{i}}
= ∑_{i∈I_{k}∩B} e^{a_{i}^{T} ȳ+λa_{i}^{T} y^{∗}−c_{i}} + ∑_{i∈I_{k}∩N} e^{a_{i}^{T} ȳ+λa_{i}^{T} y^{∗}−c_{i}}
= ∑_{i∈I_{k}∩B} e^{a_{i}^{T} ȳ−c_{i}} + ∑_{i∈I_{k}∩N} e^{a_{i}^{T} ȳ−c_{i}−λs_{i}^{∗}}
= g_{k}^{B}(ȳ) + ∑_{i∈I_{k}∩N} e^{a_{i}^{T} ȳ−c_{i}−λs_{i}^{∗}} ,

which means that

lim_{λ→+∞} g_{k}(y) = g_{k}^{B}(ȳ) ≤ 1 for all k ∈ K ,

since s_{i}^{∗} > 0 for all i ∈ N implies that all the exponents in the second sum are tending to −∞. Moreover, the objective value b^{T} y is equal to b^{T} ȳ + λb^{T} y^{∗} = b^{T} ȳ for all values of λ, since b^{T} y^{∗} = 0.

Until now, our proof has followed the lines of the corresponding proof for l_{p}-norm optimization (Theorem 4.7). However, an additional difficulty arises in the case of geometric optimization. Namely, our vector y is not necessarily feasible for problem (GP) (we may have g_{k}^{B}(ȳ) = 1 for some k and thus g_{k}(y) > 1 for all λ), and cannot therefore help us in proving that its optimum objective value is equal to b^{T} ȳ. We have to use a second trick, namely to ”mix” y with a feasible solution to make it feasible.

Let y^{0} a feasible solution to problem (GP). We know thus that

g_{k}(y^{0}) = g_{k}^{B}(y^{0}) + g_{k}^{N} (y^{0}) ≤ 1 ,

which implies

g_{k}^{B}(y^{0}) < 1

since g_{k}^{N} (y^{0}) is strictly positive. Considering now the vector y = δy^{0} + (1 − δ)ȳ + λy^{∗}, we may write

g_{k}(y) = g_{k}^{B}(y) + g_{k}^{N} (y)
= g_{k}^{B}(δy^{0} + (1 − δ)ȳ + λy^{∗}) + g_{k}^{N} (δy^{0} + (1 − δ)ȳ + λy^{∗})
= g_{k}^{B}(δy^{0} + (1 − δ)ȳ) + g_{k}^{N} (δy^{0} + (1 − δ)ȳ + λy^{∗}) ,

this last line using again the fact that A_{B}^{T}y^{∗} = 0. We have thus

lim_{λ→+∞} g_{k}(y) = g_{k}^{B}(δy^{0} + (1 − δ)ȳ)

for the same reasons as above (exponents in g_{k}^{N} tending to −∞). Since we know that functions g_{k} are convex, we have that

g_{k}^{B}(δy^{0} + (1 − δ)ȳ) ≤ δg_{k}^{B}(y^{0}) + (1 − δ)g_{k}^{B}(ȳ) < δ + (1 − δ) = 1 ,

which finally implies

lim_{λ→+∞} g_{k}(y) < 1 .

Taking now a sufficiently large value of λ, we can ensure that g_{k}(y) < 1 for all k, i.e. that y is feasible for problem (GP). The objective value associated to such a solution is equal to

b^{T} y = δb^{T} y^{0} + (1 − δ)b^{T} ȳ + λb^{T} y^{∗} = δb^{T} y^{0} + (1 − δ)b^{T} ȳ .

Letting finally δ tend to zero, we obtain a sequence of solutions y, feasible for problem (GP), whose objective values converge to b^{T} ȳ, the optimum objective value of the reduced problem (RGP), itself equal to the optimum objective value of the dual problem (GD). This is enough to prove that the primal-dual pair of problems (GP)–(GD) has a zero duality gap.

We also have the following corollary about the subvalue p^{−} of problem (GP).

Corollary 5.1. When both problems (GP) and (GD) are feasible, the optimum objective value of problem (GP) is equal to its subvalue.

Proof. Indeed, we have in general p^{∗} ≤ p^{−} ≤ d^{∗}. Since the last theorem implies p^{∗} = d^{∗}, we obtain p^{∗} = p^{−}.

5.3.4 Summary and examples

Let us summarize the possible situations about the primal problem (GP), and give corresponding examples to show that the results obtained so far cannot be sharpened.

⋄ In the best possible situation, the dual problem has a strictly positive solution and is bounded: our primal problem is guaranteed by Theorem 5.11 to be feasible and have at least one finite optimal solution with a zero duality gap. Taking for example

A = (\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix}) , b = (\begin{matrix}1 \\ 1 \end{matrix}) , c = (\begin{matrix}0 \\ 0 \end{matrix}) and I_{1} = \{1, 2\} ,

our primal-dual pair (CGP)–(CGD) becomes

sup y_{1} + y_{2}  s.t. e^{y_{1}} + e^{y_{2}} ≤ 1
inf 0 + x_{1} log \frac{x_{1}}{x_{1} + x_{2}} + x_{2} log \frac{x_{2}}{x_{1} + x_{2}}  s.t. x_{1} = 1, x_{2} = 1 and x ≥ 0 .

The only feasible dual solution is strictly positive, giving a bounded optimum objective value d^{∗} = 2 log \frac{1}{2} = −2 log 2, and we may easily check (using Lemma 4.1) that y_{1} = y_{2} = − log 2 is the only optimum primal solution, giving also p^{∗} = −2 log 2.

⋄ In the case of an unbounded dual, the primal problem has to be infeasible because of the weak duality theorem. Choosing

A = (0 1) , b = 1, c = (\begin{matrix}−1 \\ 0 \end{matrix}) and I_{1} = \{1, 2\} ,

our primal-dual pair becomes

sup y_{1}  s.t. e^{1} + e^{y_{1}} ≤ 1
inf − x_{1} + x_{1} log \frac{x_{1}}{x_{1} + x_{2}} + x_{2} log \frac{x_{2}}{x_{1} + x_{2}}  s.t. x_{2} = 1 and x ≥ 0 .

The dual is unbounded: the feasible solution x = (λ, 1) for all λ > 0 has an objective value equal to −λ+λ log \frac{λ}{λ+1} +log \frac{1}{λ+1}, which is easily shown to tend to (−∞−1−∞) = −∞ when λ → +∞. The primal problem is obviously infeasible, as expected.

⋄ When both the primal and the dual problems are feasible but the dual does not have a strictly feasible solution, the duality gap is guaranteed by Theorem 5.13 to be equal to zero with a finite common optimal objective value, but not necessarily with attainment. Adding a third variable to our previous examples

A = (\begin{matrix}1 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 2 \end{matrix}) , b = (\begin{matrix}1 \\ 1 \\ 0 \end{matrix}) , c = (\begin{matrix}0 \\ 0 \\ 1 \end{matrix}) and I_{1} = \{1, 2, 3\} ,

our primal-dual pair becomes

sup y_{1} + y_{2}  s.t. e^{y_{1}} + e^{y_{2}} + e^{y_{2}+2y_{3}−1} ≤ 1
inf x_{3} + ∑_{i|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i=1}^{3} x_{i}}  s.t. x_{1} = 1, x_{2} + x_{3} = 1, 2x_{3} = 0 and x ≥ 0 .

The only feasible dual solution x = (1, 1, 0) has a zero component and gives d^{∗} = −2 log 2. It is not too difficult to find a sequence of primal feasible solutions tending to y = (− log 2, − log 2, −∞) that establishes that the supremum of the primal problem is also equal to p^{∗} = −2 log 2. However, this value cannot be attained: the primal constraint implies e^{y_{1}} + e^{y_{2}} < 1, which in turn can be shown to force y_{1} + y_{2} < −2 log 2 using Lemma 4.1.

⋄ Our last example will demonstrate the worst situation that can happen: a feasible bounded dual problem with an infeasible primal problem. Taking

A = (\begin{matrix}1 & 0 & −1 \\ 0 & 1 & 0 \end{matrix}) , b = (\begin{matrix}1 \\ 0 \end{matrix}) , c = (\begin{matrix}1 \\ 0 \\ −1 \end{matrix}) and I_{1} = \{1, 2\} , I_{2} = \{3\} ,

our primal-dual pair becomes (after some simplifications in the dual objective)

sup y_{1}  s.t. e^{y_{1}−1} + e^{y_{2}} ≤ 1 and e^{1−y_{1}} ≤ 1
inf x_{1} − x_{3} + x_{1} log \frac{x_{1}}{x_{1} + x_{2}}  s.t. x_{1} − x_{3} = 1, x_{2} = 0 and x ≥ 0 .

All the feasible dual solution have at least one zero component and it is not difficult to compute that d^{∗} = 1 (when x = (1, 0, 0), for example). It is also easy to check that the primal problem is infeasible: the first constraint implies e^{y_{1}−1} < 1 and thus y_{1} < 1, while the second constraint forces y_{1} ≥ 1. However, Theorem 5.12 tells us that the primal problem has a subvalue p^{−} equal to d^{∗}. Indeed, relaxing the primal problem to

sup y_{1} s.t. e^{y_{1}−1} + e^{y_{2}} ≤ e^{δ} and e^{1−y_{1}} ≤ e^{δ}

for any δ > 0, we find y_{1} < 1 + δ and y_{1} ≥ 1 − δ, implying 1 − δ ≤ p̄(δ) < 1 + δ and leading to a subvalue p^{−} equal to 1, as expected.

5.4 Concluding remarks

5.4.1 Original formulation

In Subsection 5.3.1, we presented a conic formulation for the primal-dual pair of geometric optimization problems (GP)–(GD) involving linear objective functions, which allowed us to derive several duality theorems. However, the traditional formulation of geometric optimization usually involves a posynomial objective function, as in (OGP) or in (OGP’), its convexified variant. In this subsection, we show that such problems can be cast as problems with a linear objective, and outline how these duality results can be translated into this traditional formulation.

Let us restate for convenience the convexified problem (OGP’)

inf g_{0}(y) s.t. g_{k}(y) ≤ 1 for all k ∈ K \ {0} ,  (OGP’)

which is readily seen to be equivalent to

inf e^{−y_{0}} s.t. g_{0}(y) ≤ e^{−y_{0}} and g_{k}(y) ≤ 1 for all k ∈ K \ {0} ,

introducing a new variable y_{0} to express the posynomial objective. Noticing that minimizing e^{−y_{0}} amounts to maximizing y_{0}, we can rewrite this last problem as

sup y_{0} s.t. e^{y_{0}}g_{0}(y) ≤ 1 and g_{k}(y) ≤ 1 for all k ∈ K \ {0} ,

which can now be expressed in the format of (GP) as

sup b̃^{T} ỹ s.t. g̃_{k}(ỹ) ≤ 1 for all k ∈ K ,

where vector of variables ỹ ∈ ℝ^{m+1}, objective vector b̃ ∈ ℝ^{m+1} and posynomials g̃_{k} are defined by

ỹ = (\begin{matrix}y_{0} \\ y \end{matrix}) , b̃ = (\begin{matrix}1 \\ 0 \end{matrix}) , g̃_{0}(ỹ) = e^{y_{0}}g_{0}(y) and g̃_{k}(ỹ) = g_{k}(y) for all k ∈ K \ \{0\} .

This last definition of posynomials g̃_{k} corresponds to the following choice of column vectors ã_{i} (constants c_{i} are left unchanged):

ã_{i} = (\begin{matrix}1 \\ a_{i} \end{matrix}) for all i ∈ I_{0} and ã_{i} = (\begin{matrix}0 \\ a_{i} \end{matrix}) for all i ∈ I \ I_{0} .

It is now easy to find a dual for problem (OGP’), based on the known dual for (GP) and our special choice of ã_{i} and b̃. Defining a matrix Ã whose columns are the a_{i}’s, i.e.

Ã = \binom{1, …, 1 0, …, 0}{A} ,^{^{I_{0}} ^{I_{k} ∀k≠0}}

we find the dual problem

inf c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} s.t. Ãx = b̃ and x ≥ 0

or, equivalently,

inf c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} s.t. Ax = 0 , ∑_{i∈I_{0}} x_{i} = 1 and x ≥ 0 .

We can manipulate further the second part of the objective function

∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} = ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0}(x_{i} log x_{i} − x_{i} log ∑_{i∈I_{k}} x_{i})
= ∑_{i∈I} x_{i} log x_{i} − ∑_{k∈K}(∑_{i∈I_{k}} x_{i}) log(∑_{i∈I_{k}} x_{i}) ,

with the convention that 0 log 0 = 0, and find

inf c^{T} x + ∑_{i∈I} x_{i} log x_{i} − ∑_{k∈K\{0}}(∑_{i∈I_{k}} x_{i}) log(∑_{i∈I_{k}} x_{i}) s.t. Ax = 0 , ∑_{i∈I_{0}} x_{i} = 1 and x ≥ 0

(we could remove the term for k = 0 in the second sum because of the linear constraint ∑_{i∈I_{0}} x_{i} = 1). Noting finally that the objective of (OGP’) is actually e^{−y_{0}} and not y_{0}, we find after some easy transformations the final dual problem (using c_{i} = − log C_{i})

sup ∏_{i∈I} (\frac{C_{i}}{x_{i}} )^{x_{i}} ∏_{k∈K\\{0\}}(∑_{i∈I_{k}} x_{i})^{∑_{i∈I_{k}} x_{i}} s.t. Ax = 0 , ∑_{i∈I_{0}} x_{i} = 1 and x ≥ 0 . (OGD’)

This dual problem is identical to the usual formulation that can be found in the literature [DPZ67, Chapter III]. To close this discussion, we give a few hints on how to establish links between the classical theory elaborated in [DPZ67] and the results presented in Subsections 5.3.2 and 5.3.3.

The main lemma in [DPZ67, Chapter IV] is essentially our weak duality theorem with its associated set of orthogonality conditions. The first and second duality theorems from [DPZ67, Chapter III] are basically coming from Theorems 5.10 and 5.11, i.e. the application of the strong duality theorem to the primal and the dual problems (note that the hypotheses of the first duality theorem suppose primal attainment while our version only requires primal boundedness, which is a weaker condition). We also note that the notion of subinfimum in [DPZ67, Chapter VI] for the primal problem is equivalent to our concept of subvalue. Finally, the strong duality theorems in [DPZ67, Chapter VI] are closely related to our Theorem 5.13, stating that a nonzero duality gap cannot occur ; the notion of canonical problem that is heavily used in the associated proofs corresponds to the case N = ∅ in the optimal partition of problem (BLP), i.e. existence of a strictly feasible dual solution.

5.4.2 Conclusions

In this chapter, we have shown how to use the duality theory of conic optimization to derive results about geometric optimization. This process involved the introduction of a dedicated pair of convex cones G^{n} and (G^{n})^{∗}. We would like to point out that conic optimization had so far been mostly applied to self-dual cones, i.e. to linear, second-order cone and semidefinite optimization. We hope to have demonstrated here that this theory can be equally useful in the case of a less symmetric duality.

The results we obtained can be classified into two distinct categories: most of them are direct consequences of the convex nature of geometric optimization (weak and strong duality theorems), while some of them are specific to this class of problems (absence of a duality gap). The set of problems we studied differed in fact slightly from the classical formulation of geometric optimization, because of the linear objective function.

We would like to point out that this variation in the formulation was necessary since conic optimization cannot be applied directly to geometric optimization problems cast in the traditional form. Indeed, problem (OGP) is not convex, which already prevents us from applying Lagrange duality, while the pair of problems (OGP’)–(OGD’) does not feature a linear objectives and hence is not suitable for a conic formulation. However, extension of our results to the case of a posynomial objective function is straightforward, as outlined in Subsection 5.4.1. We also consider the results associated to our formulation more natural than their traditional counterparts. For example, looking at the structure of the linear constraints in the dual problem (OGD’), we understand that the presence of the normalizing constraint ∑_{i∈I_{0}} x_{1} = 1 in (OGD’) is essentially a consequence of the posynomial objective, while our dual problem (GD) features a simpler set of linear constraints Ax = b.

The proofs presented in this chapter possess in our opinion several advantages over the classical ones: in addition to being shorter, they allow us to confine the specificity of the class of problems under study to the convex cones used in the formulation. Moreover, the reason why geometric optimization has better duality properties than a general conic problem becomes clear: this is essentially due to the existence of a strictly feasible dual solution. Indeed, even if such an interior solution does not always exist, a regularization procedure involving an equivalent reduced problem can always be carried out and allows us to prove the absence of a duality gap in all cases (we note however that the property of primal attainment, satisfied when there exists a strictly feasible dual solution, is lost in this process and is thus no longer valid in the general case).

Duality for geometric optimization is a little weaker than for l_{p}-norm optimization. Namely, we do not have the primal attainment property of Theorem 4.7. The reason for this became clear in the proof of Theorem 5.13: because the solutions of the restricted primal problem were not necessarily feasible for the original primal problem, we had to perturb them with a feasible solution. Decreasing the size of this perturbation term led to a sequence of feasible solutions y, whose objective values tended to the optimal objective value of problem, but attainment was lost with this procedure since this sequence does not necessarily have a finite limit point. Indeed, the third example in Section 5.3.4 demonstrates a situation when such a sequence of feasible points tending to optimality has one component tending to +∞.

A last advantage of our conic formulation is that it allows us to benefit with minimal work from the theory of polynomial interior-point methods for convex optimization developed in [NN94]. Indeed, finding a computable self-concordant barrier for our geometric cone G^{n}, would be all that is needed to build an algorithm able to solve a geometric optimization problem up to a given accuracy within a polynomial number of arithmetic operations. However, the definition of cone G^{n} is not convenient and Chapter 6 will provide an alternative cone suitable for geometric optimization, which will prove much more suitable for the purpose of finding a self-concordant barrier.

A different cone for geometric optimization

Chapters 4 and 5 have presented a new way of formulating two classical classes of structured convex problems, l_{p}-norm and geometric optimization, using dedicated convex cones. This approach has some advantages over the traditional formulation: it simplifies the proofs of the well-known associated duality properties (i.e. weak and strong duality) and the design of a polynomial algorithm becomes straightforward.

In this chapter, we make a step towards the description of a common framework that would include these two classes of problems. Indeed, we introduce a variant of the cone for geometric optimization G^{n} used in Chapter 5 and show it is equally suitable to formulate this class of problems. This new cone has the additional advantage of being very similar to the cone L^{p} used for l_{p}-norm optimization 4, which opens the way to a common generalization.

6.1 Introduction

In Chapter 5, we defined an appropriate convex cone that allowed us to express geometric optimization problems as conic programs, the aim being to apply the general duality theory for conic optimization from Chapter 3 to these problems and prove in a seamless way the various well-known duality theorems of geometric optimization. The goal of this chapter is to introduce a variation of this convex cone that preserves its ability to model geometric optimization problems but bears more resemblance with the cone that was introduced for l_{p}-norm optimization in Chapter 4, hinting for a common generalization of these two families of cones.

This chapter is organized as follows: Section 6.2 introduces the convex cones needed to model geometric optimization and studies some of their properties. Section 6.4 constitutes the main part of this chapter and demonstrates how the above-mentioned cones enable us to model primal and dual geometric optimization problems in a seamless fashion. Modelling the primal problem with our first cone is rather straightforward and writing down its dual is immediate, but some work is needed to prove the equivalence with the traditional formulation of a dual geometric optimization problem. Finally, concluding remarks in Section 6.5 provide some insight about the relevance of our approach and pave the way to Chapter 7, where it is applied to a much larger class of cones.

6.2 The extended geometric cone

Let us introduce the extended geometric cone G_{2}^{n}, which will allow us to give a conic formulation of geometric optimization problems.

Definition 6.1. Let n ∈ ℕ. The extended geometric cone G_{2}^{n} is defined by

G_{2}^{n} = \{(x, θ, κ) ∈ ℝ_{+}^{n} × ℝ_{+} × ℝ_{+} | θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ κ\}

using in the case of a zero denominator the following convention:

e^{− \frac{x_{i}}{0}} = 0 .

We observe that this convention results in (x, 0, κ) ∈ G_{2}^{n} for all x ∈ ℝ_{+}^{n} and κ ∈ ℝ_{+}. As a special case, we mention that G_{2}^{0} is the 2-dimensional nonnegative orthant ℝ_{+}^{2}. The main difference between this cone and the original geometric cone G^{n} described in Chapter 5 is the addition of a variable κ.

In order to use the conic formulation from Chapter 3, we first prove that G_{2}^{n} is a convex cone.

Theorem 6.1. G_{2}^{n} is a convex cone.

Proof. Let us first introduce the following function

f_{n} : ℝ_{+}^{n} × ℝ_{+} ↦ ℝ_{+} : (x, θ) ↦ ∑_{i=1}^{n} θe^{− \frac{x_{i}}{θ}} .

With the convention mentioned above, its effective domain is ℝ_{+}^{n+1}. It is straightforward to check that f_{n} is positively homogeneous, i.e. f_{n}(λx, λθ) = λf_{n}(x, θ) for λ ≥ 0. Moreover, f_{n} is subadditive, i.e. f_{n}(x + x′, θ + θ′) ≤ f_{n}(x, θ) + f_{n}(x′, θ′). In order to show this property, we can work on each term of the sum separately, which means that we only need to prove the following inequality for all x, x′ ∈ ℝ and θ, θ′ ∈ ℝ_{+}:

θe^{− \frac{x}{θ}} + θ′e^{− \frac{x′}{θ′}} ≥ (θ + θ′)e^{− \frac{x+x′}{θ+θ′}} .

First observe that this inequality holds when θ = 0 or θ′ = 0. For example, when θ = 0, we have to check that θ′e^{− \frac{x′}{θ′}} ≥ θ′e^{− \frac{x+x′}{θ′}}, which is a consequence of the fact that x ↦ e^{−x} is a decreasing function. When θ + θ′ > 0, we use the well-known fact that x ↦ e^{−x} is a convex function on ℝ_{+}, implying that λe^{−a} + λ′e^{−a′} ≥ e^{−(λa+λa′)} for any nonnegative a, a′, λ and λ′ satisfying λ + λ′ = 1. Choosing a = \frac{x}{θ}, a′ = \frac{x′}{θ′}, λ = \frac{θ}{θ+θ′} and λ′ = \frac{θ′}{θ+θ′}, we find that

\frac{θ}{θ+θ′} e^{− \frac{x}{θ}} + \frac{θ′}{θ+θ′} e^{− \frac{x′}{θ′}} ≥ e^{− \frac{θ}{θ+θ′} \frac{x}{θ} − \frac{θ′}{θ+θ′} \frac{x′}{θ′}} ,

which, after multiplying by (θ + θ′), lead to the desired inequality

θe^{− \frac{x}{θ}} + θ′e^{− \frac{x′}{θ′}} ≥ (θ + θ′)e^{− \frac{x+x′}{θ+θ′}} .

Positive homogeneity and subadditivity imply that f_{n} is a convex function. Since f_{n}(x, θ) ≥ 0 for all x ∈ ℝ_{+}^{n} and θ ∈ ℝ_{+}, we notice that G_{2}^{n} is the epigraph of f_{n}, i.e.

epi f_{n} = {(x, θ, κ) ∈ ℝ_{+}^{n} × ℝ_{+} × ℝ | f_{n}(x, θ) ≤ κ} = G_{2}^{n} .

G_{2}^{n} is thus the epigraph of a convex positively homogeneous function, hence a convex cone [Roc70a].

Note that the above proof bears much more resemblance with the corresponding proof for the L^{p} cone of l_{p}-norm optimization than the original geometric cone G^{n}. We now proceed to prove some properties of the extended geometric cone G_{2}^{n}.

Theorem 6.2. G_{2}^{n} is closed.

Proof. Let {(x^{k}, θ^{k}, κ^{k})} a sequence of points in ℝ_{+}^{n+2} such that (x^{k}, θ^{k}, κ^{k}) ∈ G_{2}^{n} for all k and lim_{k→∞}(x^{k}, θ^{k}, κ^{k}) = (x^{∞}, θ^{∞}, κ^{∞}). In order to prove that G_{2}^{n} is closed, it suffices to show that (x^{∞}, θ^{∞}, κ^{∞}) ∈ G_{2}^{n}. Let us distinguish two cases:

⋄ θ^{∞} > 0. Using the easily proven fact that functions (x_{i}, θ) ↦ θe^{− \frac{x_{i}}{θ}} are continuous on ℝ_{+} × ℝ_{++}, we have that

θ^{∞} ∑_{i=1}^{n} e^{− \frac{x^{∞}_{i}}{θ^{∞}}} = ∑_{i=1}^{n} θ^{∞}e^{− \frac{x^{∞}_{i}}{θ^{∞}}} = ∑_{i=1}^{n} lim_{k→∞} θ^{k}e^{− \frac{x^{k}_{i}}{θ^{k}}} = lim_{k→∞} ∑_{i=1}^{n} θ^{k}e^{− \frac{x^{k}_{i}}{θ^{k}}} ≤ lim_{k→∞} κ^{k} = κ^{∞} ,

which implies (x^{∞}, θ^{∞}) ∈ G_{2}^{n}.

⋄ θ^{∞} = 0. Since (x^{k}, θ^{k}, κ^{k}) ∈ G_{2}^{n}, we have x^{k} ≥ 0 and κ^{k} ≥ 0, which implies that x^{∞} ≥ 0 and k^{∞} ≥ 0. This shows that (x^{∞}, 0, κ^{∞}) ∈ G_{2}^{n}.

In both cases, (x^{∞}, θ^{∞}, κ^{∞}) is shown to belong to G_{2}^{n}, which proves the claim.

It is also interesting to identify the interior of this cone.

Theorem 6.3. The interior of G_{2}^{n} is given by

int G_{2}^{n} = \{(x, θ, κ) ∈ ℝ_{++}^{n} × ℝ_{++} × ℝ_{++} | θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} < κ\} .

Proof. According to Lemma 7.3 in [Roc70a] we have

int G_{2}^{n} = int epi f_{n} = {(x, θ, κ) | (x, θ) ∈ int dom f_{n} and f_{n}(x, θ) < κ} .

The above-stated result then simply follows from the fact that int dom f_{n} = ℝ_{++}^{n+1}.

Corollary 6.1. The cone G_{2}^{n} is solid.

Proof. It suffices to prove that there exists at least one point that belongs to int G_{2}^{n} (Definition 3.3). Taking for example the point (e, \frac{1}{n}, 1), where e stands for the n-dimensional all-one vector, we have

∑_{i=1}^{n} θe^{− \frac{x_{i}}{θ}} = e^{−n} < 1 = κ ,

and therefore (e, \frac{1}{n}, 1) ∈ int G_{2}^{n}.

We also have the following fact:

Theorem 6.4. G_{2}^{n} is pointed.

Proof. The fact that 0 ∈ G_{2}^{n} ⊆ ℝ_{+}^{n+2} implies that G_{2}^{n} ∩ −G_{2}^{n} = {0}, i.e. G_{2}^{n} is pointed (Definition 3.2).

To summarize, G_{2}^{n} is a solid pointed closed convex cone, hence suitable for conic optimization.

6.3 The dual extended geometric cone

In order to express the dual of a conic problem involving the extended geometric cone G_{2}^{n}, we need to find an explicit description of its dual.

Theorem 6.5. The dual of G_{2}^{n} is given by

(G_{2}^{n})^{∗} = \{(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ_{+}^{n} × ℝ × ℝ_{+} | θ^{∗} ≥ ∑_{0<x_{i}^{∗}<κ^{∗}}(x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} − x_{i}^{∗}) − ∑_{x_{i}^{∗}≥κ^{∗}} κ^{∗}\} .

Proof. Using Definition 3.4 for the dual cone, we have

(G_{2}^{n})^{∗} = {(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ^{n} × ℝ × ℝ | (x, θ, κ)^{T} (x^{∗}, θ^{∗}, κ^{∗}) ≥ 0 for all (x, θ, κ) ∈ G_{2}^{n}}

(the ^{∗} superscript on variables x^{∗} and θ^{∗} is a reminder of their dual nature). We first note that in the case θ = 0, we may choose any x ∈ ℝ_{+}^{n} and κ ∈ ℝ_{+} and have (x, 0, κ) ∈ G_{2}^{n}, which means that the product

(x, θ, κ)^{T} (x^{∗}, θ^{∗}, κ^{∗}) = x^{T} x^{∗} + θθ^{∗} + κκ^{∗} = x^{T} x^{∗} + κκ^{∗}

has to be nonnegative for all (x, κ) ∈ ℝ_{+}^{n+1} and is easily seen to imply that x^{∗} and κ^{∗} are nonnegative. We may now suppose θ > 0, (x^{∗}, κ^{∗}) ≥ 0 and write

x^{T} x^{∗} + θθ^{∗} + κκ^{∗} ≥ 0 for all (x, θ, κ) ∈ G_{2}^{n}
⇔ x^{T} x^{∗} + θθ^{∗} + (θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} )κ^{∗} ≥ 0 for all (x, θ) ∈ ℝ_{+}^{n} × ℝ_{++}

⇔ θ^{∗} ≥ −\frac{x^{T} x^{∗}}{θ} − κ^{∗} ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} for all (x, θ) ∈ ℝ_{+}^{n} × ℝ_{++}

⇔ θ^{∗} ≥ −t^{T} x^{∗} − κ^{∗} ∑_{i=1}^{n} e^{−t_{i}} for all t ∈ ℝ_{+}^{n}

⇔ θ^{∗} ≥ − ∑_{i=1}^{n}(t_{i}x_{i}^{∗} + κ^{∗}e^{−t_{i}}) for all t ∈ ℝ_{+}^{n} ,

where we have defined t_{i} = \frac{x_{i}}{θ} for convenience. We now proceed to seek the greatest possible lower bound on θ^{∗}, examining each term of the sum separately: we have thus to seek the minimum of

t_{i}x_{i}^{∗} + κ^{∗}e^{−t_{i}} .

The derivative of this quantity with respect to t_{i} being equal to x_{i}^{∗} − κ^{∗}e^{−t_{i}}, we have a minimum when t_{i} = − log \frac{x_{i}^{∗}}{κ^{∗}}, but we have to take into account the fact that t_{i} has to be nonnegative, which leads us to distinguish the following three cases

⋄ κ^{∗} = 0: in this case, the minimum is always equal to 0,

⋄ κ^{∗} > 0 and x_{i}^{∗} ≤ κ^{∗}: in this case, the minimum is attained for a nonnegative t_{i} and is equal to −x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} +x_{i}^{∗}, this quantity being taken as equal to zero in the case of x_{i}^{∗} = 0,

⋄ κ^{∗} > 0 and x_{i}^{∗} > κ^{∗}: in this case, the minimum value for a nonnegative t is attained for t = 0 and is equal to κ^{∗}.

These three cases can be summarized with

inf_{t_{i}≥0}(t_{i}x_{i}^{∗} + κ^{∗}e^{−t_{i}}) = \begin{cases}−x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} + x_{i}^{∗} & when x_{i}^{∗} < κ^{∗} \\ κ^{∗} & when x_{i}^{∗} ≥ κ^{∗} \end{cases} .

Since all of these lower bounds can be simultaneously attained with a suitable choice of t, we can state the final defining inequalities of our dual cone as

x^{∗} ≥ 0, κ^{∗} ≥ 0 and θ^{∗} ≥ ∑_{0<x_{i}^{∗}<κ^{∗}}(x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} − x_{i}^{∗}) − ∑_{x_{i}^{∗}≥κ^{∗}} κ^{∗} .

As a special case, since G_{2}^{0} = ℝ_{+}^{2}, we check that (G_{2}^{0})^{∗} = (ℝ_{+}^{2})^{∗} = ℝ_{+}^{2}, as expected.

Note 6.1. It can be easily checked that the lower bound on θ^{∗} appearing in the definition is always nonpositive, which means that (x^{∗}, θ^{∗}, κ^{∗}) ∈ (G_{2}^{n})^{∗} as soon as x^{∗} and θ^{∗} are nonnegative. This fact could have been guessed prior to any computation: noticing that G_{2}^{n} ⊆ ℝ_{+}^{n+2} and (ℝ_{+}^{n+2})^{∗} = ℝ_{+}^{n+2}, we immediately have that (G_{2}^{n})^{∗} ⊇ ℝ_{+}^{n+2}, because taking the dual of a set inclusion reverses its direction.

Finding the dual of G_{2}^{n} was a little involved, but establishing its properties is straightforward.

Theorem 6.6. (G_{2}^{n})^{∗} is a solid, pointed, closed convex cone. Moreover, ((G_{2}^{n})^{∗})^{∗} = G_{2}^{n}.

Proof. The proof of this fact is immediate by Theorem 3.3 since (G_{2}^{n})^{∗} is the dual of a solid, pointed, closed convex cone.

The interior of (G_{2}^{n})^{∗} is also rather easy to obtain:

Theorem 6.7. The interior of (G_{2}^{n})^{∗} is given by

int(G_{2}^{n})^{∗} = \{(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ_{++}^{n} × ℝ × ℝ_{++} | θ^{∗} > ∑_{0<x_{i}^{∗}<κ^{∗}}(x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} − x_{i}^{∗}) − ∑_{x_{i}^{∗}≥κ^{∗}} κ^{∗}\} .

Proof. We first note that (G_{2}^{n})^{∗}, a convex set, is the epigraph of the following function

f_{n} : ℝ_{+}^{n} × ℝ_{+} ↦ ℝ : (x^{∗}, κ^{∗}) ↦ ∑_{0<x_{i}^{∗}<κ^{∗}}(x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} − x_{i}^{∗}) − ∑_{x_{i}^{∗}≥κ^{∗}} κ^{∗} ,

which implies that f_{n} is convex (by definition of a convex function). Hence we can apply Lemma 7.3 from [Roc70a] to get

int(G_{2}^{n})^{∗} = int epi f_{n} = {(x^{∗}, κ^{∗}, θ^{∗}) ∈ int dom f_{n} × ℝ | θ^{∗} > f_{n}(x^{∗}, κ^{∗})} ,

which is exactly our claim since int(ℝ_{+}^{n} × ℝ_{+}) = ℝ_{++}^{n} × ℝ_{++}.

6.4 A conic formulation

This is the main section of this chapter, where we show how a primal-dual pair of geometric optimization problems can be modelled using the G_{2}^{n} and (G_{2}^{n})^{∗} cones.

6.4.1 Modelling geometric optimization

Let us restate here for convenience the definition of the standard primal geometric optimization problem.

sup b^{T} y s.t. g_{k}(y) ≤ 1 for all k ∈ K ,  (GP)

where functions g_{k} are defined by

g_{k} : ℝ^{m} ↦ ℝ_{++} : y ↦ ∑_{i∈I_{k}} e^{a_{i}^{T} y−c_{i}} .

We first introduce a vector of auxiliary variables s ∈ ℝ^{n} to represent the exponents used in functions g_{k}, more precisely we let

s_{i} = c_{i} − a_{i}^{T} y for all i ∈ I or, in matrix form, s = c − A^{T} y ,

where A is a m × n matrix whose columns are a_{i}. Our problem becomes then

sup b^{T} y s.t. s = c − A^{T} y and ∑_{i∈I_{k}} e^{−s_{i}} ≤ 1 for all k ∈ K ,

which is readily seen to be equivalent to the following, using the definition of G_{2}^{n} (where both variables κ and θ have been fixed to 1),

sup b^{T} y s.t. A^{T} y + s = c and (s_{I_{k}}, 1, 1) ∈ G_{2}^{#I_{k}} for all k ∈ K ,

and finally to

sup b^{T} y s.t. (\begin{matrix}A^{T} \\ 0 \\ 0 \end{matrix} ) y + (\begin{matrix}s \\ v \\ w \end{matrix}) = (\begin{matrix}c \\ e \\ e \end{matrix}) and (s_{I_{k}}, v_{k}, w_{k}) ∈ G_{2}^{n_{k}} for all k ∈ K , (CG_{2}P)

where e is the all-one vector in ℝ^{r}, n_{k} = #I_{k} and two additional vectors of fictitious variables v, w ∈ ℝ^{r} have been introduced, whose components are fixed to 1 by part of the linear constraints. This is exactly a conic optimization problem, in the dual form (CD), using variables (ỹ, s̃), data (Ã, b̃, c̃) and a cone K^{∗} such that

ỹ = y, s̃ = (\begin{matrix}s \\ v \\ w \end{matrix}) , Ã = (A 0 0) , b̃ = b, c̃ = (\begin{matrix}c \\ e \\ e \end{matrix}) and K^{∗} = G_{2}^{n_{1}} × G_{2}^{n_{2}} × ⋯ × G_{2}^{n_{r}} ,

where K^{∗} has been defined as the Cartesian product of several disjoint extended geometric cones, according to Note 3.1, in order to deal with multiple conic constraints involving disjoint sets of variables. We also note that the fact that we have been able to model geometric optimization with a convex cone is a proof that these problems are convex.

6.4.2 Deriving the dual problem

Using properties of G_{2}^{n} and (G_{2}^{n})^{∗} proved in the previous section, it is straightforward to show that K^{∗} is a solid, pointed, closed convex cone whose dual is

(K^{∗})^{∗} = K = (G_{2}^{n_{1}})^{∗} × (G_{2}^{n_{2}})^{∗} × ⋯ × (G_{2}^{n_{r}})^{∗} ,

another solid, pointed, closed convex cone, according to Theorem 3.3. This allows us to derive a dual problem to (CG_{2}P) in a completely mechanical way and find the following conic optimization problem, expressed in the primal form (CP):

inf (\begin{matrix}c \\ e \\ e \end{matrix})^{T} (\begin{matrix}x \\ z \\ u \end{matrix}) s.t. (A 0 0) (\begin{matrix}x \\ z \\ u \end{matrix}) = b and (x_{I_{k}}, z_{k}, u_{k}) ∈ (G_{2}^{n_{k}})^{∗} ∀k ∈ K , (CG_{2}D)

where x ∈ ℝ^{n}, z ∈ ℝ^{r} and u ∈ ℝ^{r} are the vectors we optimize. This problem can be simplified: making the conic constraints explicit, we find

inf c^{T} x + e^{T} z + e^{T} u s.t. \begin{cases}Ax = b, x_{I_{k}} ≥ 0, u_{k} ≥ 0 , \\ z_{k} ≥ ∑_{i∈I_{k}|0<x_{i}<u_{k}}(x_{i} log \frac{x_{i}}{u_{k}} − x_{i}) − ∑_{i∈I_{k}|x_{i}≥u_{k}} u_{k} ∀k ∈ K , \end{cases}

which can be further reduced to

inf c^{T} x+e^{T} u+ ∑_{k∈K}( ∑_{i∈I_{k}|0<x_{i}<u_{k}}(x_{i} log \frac{x_{i}}{u_{k}} −x_{i})− ∑_{i∈I_{k}|x_{i}≥u_{k}} u_{k}) s.t. Ax = b, u ≥ 0 and x ≥ 0 .

Indeed, since each variable z_{k} is free except for the inequality coming from the associated conic constraint, these inequalities must be satisfied with equality at each optimum solution and variables z can therefore be removed from the formulation. At this point, the formulation we have is simpler than the pure conic dual but is still different from the usual geometric optimization dual problem (GD) one can find in the literature. A little bit of calculus will help us to bridge the gap: let us fix k and consider the corresponding terms in the objective

c_{I_{k}}^{T}x_{I_{k}} + u_{k} + ∑_{i∈I_{k}|0<x_{i}<u_{k}}(x_{i} log \frac{x_{i}}{u_{k}} − x_{i}) − ∑_{i∈I_{k}|x_{i}≥u_{k}} u_{k} .

We would like to eliminate variable u_{k}, i.e. find for which value of u_{k} the previous quantity is minimum. It is first straightforward to check that such a value of u_{k} must satisfy x_{i} < u_{k} for all i ∈ I_{k}, i.e. will only involve the first summation sign (since the value −u_{k} in the second sum is attained as a limit case in the first sum when x_{i} tends to u_{k} from below). Taking the derivative with respect to u_{k} and equating it to zero we find

0 = 1 + ∑_{i∈I_{k}} x_{i} \frac{u_{k}}{x_{i}} (−\frac{x_{i}}{u_{k}^{2}} ) = 1 − \frac{∑_{i∈I_{k}} x_{i}}{u_{k}}, which implies u_{k} = ∑_{i∈I_{k}} x_{i} .

Our objective terms become equal to

c_{I_{k}}^{T}x_{I_{k}} + ∑_{i∈I_{k}} x_{i} + ∑_{i∈I_{k}}(x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} − x_{i}) = c_{I_{k}}^{T}x_{I_{k}} + ∑_{i∈I_{k}} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} ,

and leads to the following simplified dual problem

inf c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} s.t. Ax = b and x ≥ 0 ,  (GD)

which is, as we expected, the traditional form of a dual geometric optimization problem (see Chapter 5). This confirms the relevance of our pair of primal-dual extended geometric cones as a tool to model the class of geometric optimization problems.

6.5 Concluding remarks

In this chapter, we have formulated geometric optimization problems in a conic way using some suitably defined convex cones G_{2}^{n} and (G_{2}^{n})^{∗}. This approach has the following advantages:

⋄ Classical results from the standard conic duality theory can be applied to derive the duality properties of a pair of geometric optimization problems, including weak and strong duality. This was done in Chapters 4 and 5 and could be done here in a very similar fashion.

⋄ Proving that geometric optimization problems can be solved in polynomial time can now be done rather easily: finding a suitable (i.e. computable) self-concordant barrier for cones G_{2}^{n} and (G_{2}^{n})^{∗} is essentially all that is needed.

⋄ Unlike the cones G^{n} and (G^{n})^{∗} introduced in Chapter 5, the pair of cones we have introduced in this chapter bears some strong similarities with the cones L^{p} and L_{s}^{q} used in Chapters 4 for l_{p}-norm optimization. We can indeed write the following equivalent definition of the cone L^{p}

L^{p} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ_{+} | θ ∑_{i=1}^{n} \frac{1}{p_{i}} |\frac{x_{i}}{θ} |^{p_{i}} ≤ κ\}

and compare it to

G_{2}^{n} = \{(x, θ, κ) ∈ ℝ_{+}^{n} × ℝ_{+} × ℝ_{+} | θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ κ\} .

The only difference between those two definitions is the function that is applied to the quantities \frac{x_{i}}{θ} for each term of the sum: the extended geometric cone G_{2}^{n} uses x ↦ e^{−x} while the l_{p}-norm cone L^{p} is based on x ↦ \frac{1}{p_{i}} |x|^{p_{i}}. This observation is the first step towards the design of a common framework that would encompass geometric optimization, l_{p}-norm optimization and several other kinds of structured convex problems, which is the topic of Chapter 7.

A general framework for separable convex optimization

In this chapter, we introduce the notion of separable cone K^{f} to generalize the cones L^{p} and G_{2}^{n} presented in Chapters 4 and 6 to model l_{p}-norm and geometric optimization. We start by giving a suitable definition for this new class of cones, and then proceed to investigate their properties and compute the corresponding dual cones, which share the same structure as their primal counterparts. Special care is taken to handle in a correct manner the boundary of these cones.

This allows us to present a new class of primal-dual convex problems using the conic formulation of Chapter 3, with the potential to model many different types of constraints.

7.1 Introduction

Chapter 4 and Chapter 5 were devoted to the study of l_{p}-norm optimization and geometric optimization using a conic formulation. The reader has probably noticed a lot of similarity between these two chapters. Indeed, in both cases, we started by defining an ad hoc convex cone, studied its properties (i.e. proved closedness, solidness, pointedness and identified its interior), computed the corresponding dual cone and listed the associated orthogonality conditions.

The primal cone allowed us to model the traditional primal formulation of these two classes of problems, while the dual cone allowed us to find in a straightforward manner the classical dual associated to these problems. Furthermore, this setting allowed us to prove the associated duality properties (using the theory of conic duality, see Chapter 3) and in the case of l_{p}-norm optimization to describe an interior-point polynomial-time algorithm (using the framework of self-concordant barriers, see Chapter 2). This new approach had the advantage of simplifying the proofs and giving some insight on the duality properties of these two classes of problems, which are better than in the case of a general convex problem.

The purpose of this chapter is to show that this process can be generalized to a great extent. Indeed, Chapter 6 started to bridge the gap between l_{p}-norm optimization and geometric optimization by giving an alternate formulation for the latter. We recall here the last remark of Section 6.5, whose purpose was to compare the following equivalent definition of the cone L^{p}

L^{p} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ_{+} | θ ∑_{i=1}^{n} \frac{1}{p_{i}} |\frac{x_{i}}{θ} |^{p_{i}} ≤ κ\}

with the definition of the extended geometric cone

G_{2}^{n} = \{(x, θ, κ) ∈ ℝ_{+}^{n} × ℝ_{+} × ℝ_{+} | θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ κ\} .

We noticed that the only difference between those two definitions was the function that was applied to the quantities \frac{x_{i}}{θ} for each term of the sum: the extended geometric cone G_{2}^{n} used the negative exponential x ↦ e^{−x} while the l_{p}-norm cone L^{p} was based on x ↦ \frac{1}{p_{i}} |x|^{p_{i}}. The purpose of this chapter is to generalize these two cones, based on the use of an arbitrary convex function in the definition of a cone with the same structure as L^{p} and G_{2}^{n}.

This chapter is organized as follows. In order to use the setting of conic optimization, we define in Section 7.2 a large class of convex cones called separable cones. Section 7.3 is devoted to the computation of the corresponding dual cone. Section 7.4 provides an alternate and more explicit definition of these cones. Section 7.5 shows that the class of separable cones is indeed a generalization of the l_{p}-norm and geometric cones presented in previous chapters. Section 7.6 presents the primal-dual pair of conic optimization problems built with our separable cones and finally Section 7.7 concludes with some possible directions for future research.

7.2 The separable cone

Let n ∈ ℕ and let us consider a set of n convex scalar functions

{f_{i} : ℝ ↦ ℝ ∪ {+∞} : x ↦ f_{i}(x) for all 1 ≤ i ≤ n} ,

which can be conveniently assembled into an n-dimensional function of ℝ

f : ℝ ↦ (ℝ ∪ {+∞})^{n} : x ↦ (f_{1}(x), f_{2}(x), …, f_{n}(x)) .

Function f is obviously also convex. We will also require functions f to be proper and closed, according to the following definitions (see e.g. [Roc70a]).

Definition 7.1. A convex function f : ℝ^{n} ↦ ℝ ∪ {+∞} is proper if it is not identically equal to +∞ on ℝ^{n}, i.e. if there exists at least a point y ∈ ℝ^{n} such that f(y) is finite.

Definition 7.2. A convex function f : ℝ^{n} ↦ ℝ ∪ {+∞} is closed if and only if its epigraph is closed, i.e. if

{(x, t) ∈ ℝ^{n} × ℝ | f(x) ≤ t} = cl{(x, t) ∈ ℝ^{n} × ℝ | f(x) ≤ t} .

Theorem 7.1 in [Roc70a] states that a function f is closed if and only if it is lower semi-continuous, according to the following definition:

Definition 7.3. A function f : ℝ^{n} ↦ ℝ ∪ {+∞} is lower semi-continuous if and only if

f(x) ≤ lim_{k↦+∞} f(x^{k})

for every sequence such that x^{k} converges to x and the limit of f(x^{1}), f(x^{2}), … exists in ℝ ∪ {+∞}.

Let us now consider the following set

K^{∘f} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{++} × ℝ | θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ}) ≤ κ\} .

The closure of this set will be define the separable cone K^{f}.

Definition 7.4. The separable cone K^{f} ⊆ ℝ^{n+2} is defined by

K^{f} = cl K^{∘f} = cl\{(x, θ, κ) ∈ ℝ^{n} × ℝ_{++} × ℝ | θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ}) ≤ κ\} .

Comparing this with the definitions of cones L^{p}, G^{n} and G_{2}^{n}, we notice that we did not have to introduce an arbitrary convention for the case of a zero denominator, since the definition of K^{∘f}, which is based on the potentially undefined argument \frac{x_{i}}{θ}, only uses strictly positive values of θ.

We first show that K^{f} is a closed convex cone, i.e. that it will be suitable for conic optimization.

Theorem 7.1. K^{f} is a closed convex cone.

Proof. Since K^{f} is obviously a closed set, we only have to prove that it is closed under addition and nonnegative scalar multiplication. Let us first suppose y ∈ K^{f} and consider λ > 0. Since K^{f} = cl K^{∘f}, we have that there exists a sequence y^{1}, y^{2}, … converging to y such that y^{k} ∈ K^{∘f} for all k. Letting y^{k} = (x^{k}, θ^{k}, κ^{k}), we immediately see that λy^{k} = (λx^{k}, λθ^{k}, λκ^{k}) also belongs to K^{∘f}, since

θ^{k} ∈ ℝ_{++} ⇔ λθ^{k} ∈ ℝ_{++} and θ^{k} ∑_{i=1}^{n} f_{i}(\frac{x_{i}^{k}}{θ^{k}}) ≤ κ^{k} ⇔ λθ^{k} ∑_{i=1}^{n} f_{i}(\frac{λx_{i}^{k}}{λθ^{k}}) ≤ λκ^{k} for all λ > 0 .

Taking now the limit of the sequence λy^{k}, we find that lim_{k→+∞} λy^{k} = λ lim_{k→+∞} y^{k} = λy belongs to K^{f}, because of the closure operation.

We also have to handle the case λ = 0, i.e. prove that 0 always belongs to K^{f}. Indeed, recalling that functions f_{i} are proper, we have for each index i a real x̂_{i} such that f_{i}(x̂_{i}) < +∞. This is easily seen to imply that the point (x̂, 1, ∑_{i=1}^{n} f_{i}(x̂_{i})) belongs to K^{∘f}. Using the above discussion, we immediately also have that (μx̂, μ, μ ∑_{i=1}^{n} f_{i}(x̂_{i})) ∈ K^{∘f} for all μ > 0. Letting μ tend to 0, we find that the limit point of this sequence is (0, 0, 0) and has to belong to the closure of K^{∘f}, i.e. that 0 ∈ K^{f}.

Let us now consider another point z belonging to K^{f}, which implies the existence of a sequence z^{1}, z^{2}, … converging to z such that z^{k} ∈ K^{∘f} for all k. We would like to show that y^{k} + z^{k} belongs to K^{∘f}, since it would then imply that

lim_{k→+∞}(y^{k} + z^{k}) = lim_{k→+∞} y^{k} + lim_{k→+∞} z^{k} = y + z ,

which belongs to cl K^{∘f} = K^{f}. Indeed, letting z^{k} = (x′^{k}, θ′^{k}, κ′^{k}), we first check that θ^{k}+θ′^{k} > 0. Convexity of functions f_{i} implies then that

f_{i}(\frac{x_{i}^{k} + x′_{i}^{k}}{θ^{k} + θ′^{k}} ) = f_{i}(\frac{θ^{k}}{θ^{k} + θ′^{k}} \frac{x_{i}^{k}}{θ^{k}} + \frac{θ′^{k}}{θ^{k} + θ′^{k}} \frac{x′_{i}^{k}}{θ′^{k}} ) ≤ \frac{θ^{k}}{θ^{k} + θ′^{k}} f_{i}(\frac{x_{i}^{k}}{θ^{k}} ) + \frac{θ′^{k}}{θ^{k} + θ′^{k}} f_{i}(\frac{x′_{i}^{k}}{θ′^{k}} ) ,

since we have \frac{θ^{k}}{θ^{k}+θ′^{k}} + \frac{θ′^{k}}{θ^{k}+θ′^{k}} = 1. This shows that

(θ^{k} + θ′^{k}) ∑_{i=1}^{n} f_{i}(\frac{x_{i}^{k} + x′_{i}^{k}}{θ^{k} + θ′^{k}} ) ≤ θ^{k} ∑_{i=1}^{n} f_{i}(\frac{x_{i}^{k}}{θ^{k}} ) + θ′^{k} ∑_{i=1}^{n} f_{i}(\frac{x′_{i}^{k}}{θ′^{k}} ) ≤ κ^{k} + κ′^{k} ,

i.e. that (y^{k} + z^{k}) belongs to K^{∘f}, which concludes this proof (which was quite similar to the one we used to show that L^{p} is convex).

Let us now identify the interior of the separable cone K^{f}.

Theorem 7.2. The interior of K^{f} is given by

int K^{f} = int K^{∘f}
= \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{++} × ℝ | x_{i} ∈ int dom f_{i} ∀1 ≤ i ≤ n and θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ}) < κ\} .

Proof. The first equality is obvious, since int cl S = int S for any set S. We note K^{∘f} can be seen as the epigraph of a function g defined by

g : ℝ^{n} × ℝ_{++} ↦ ℝ : (x, θ) ↦ θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ}) ,

i.e. (x, θ, κ) ∈ K^{∘f} ⇔ g(x, θ) ≤ κ. Moreover, the effective domain of g is easily seen to be equal to dom f_{1} × dom f_{2} × ⋯ × dom f_{n} × ℝ_{++}. Using now Lemma 7.3 in [Roc70a], we find that

int K^{∘f} = {(x, θ, κ) ∈ ℝ^{n} × ℝ_{++} × ℝ | x_{i} ∈ int dom f_{i} for all 1 ≤ i ≤ n and g(x, θ) < κ} ,

which is exactly what we wanted to prove.

At this point, we make an additional assumption on our scalar functions f_{i}, namely we require that int dom f_{i} ≠ ∅. Recall that properness of f_{i} only implies dom f_{i} ≠ ∅. Since we know that dom f_{i} is a convex subset in ℝ [Roc70a, p. 23], i.e. an interval, we see that the only effect of this assumption is to exclude the case where dom f_{i} = {a}, i.e. the situation where f_{i} is infinite everywhere except at a single point. With this assumption, we have that

Corollary 7.1. The separable cone K^{f} is solid.

Proof. It suffices to prove that there exists at least one point (x, θ, κ) that belongs to int K^{f}. The previous theorem shows this is trivially done by taking x_{i} ∈ int dom f_{i} for all 1 ≤ i ≤ n, θ = 1 and a sufficiently large κ.

7.3 The dual separable cone

We are now going to determine the dual cone of G^{f}. In order to do that, we have to introduce the notion of conjugate function (see e.g. [Roc70a]).

Definition 7.5. The conjugate of the convex function f : ℝ^{n} ↦ ℝ ∪ {+∞} is the function

f^{∗} : ℝ^{n} ↦ ℝ ∪ {+∞} : x^{∗} ↦ sup_{x∈ℝ^{n}} {x^{T} x^{∗} − f(x)} .

Theorem 12.2 in [Roc70a] states that the conjugate of a closed proper convex function is also closed, proper and convex, and that the conjugate of that conjugate is equal to the original function. We will require in addition that int dom f_{i}^{∗} ≠ ∅ as for functions f_{i}.

Just as we did in Chapter 4 for the L^{p} cone, it is convenient to introduce a switched separable cone K_{s}^{f}, which is obtained by taking the opposite x variables and exchanging the roles of variables θ and κ (note that in the case of the L^{p} cone, the opposite sign of the dual x^{∗} variables was hidden by the fact that the conjugate functions f_{i}^{∗} were even).

Definition 7.6. The switched separable cone K_{s}^{f} ⊆ ℝ^{n} × ℝ × ℝ_{+} is defined by

(x, θ, κ) ∈ K_{s}^{f} ⇔ (−x, κ, θ) ∈ K^{f} .

We are now ready to describe the dual of K^{f}.

Theorem 7.3. Let us define f^{∗} as

f^{∗} : ℝ ↦ (ℝ ∪ {+∞})^{n} : x ↦ (f_{1}^{∗}(x), f_{2}^{∗}(x), …, f_{n}^{∗}(x))

where f_{i}^{∗} is the scalar function that is conjugate to f_{i}. The dual of K^{f} is K_{s}^{f^{∗}}.

Proof. Using first the fact that (cl C)^{∗} = C^{∗} [Roc70a, p. 121], we have (K^{f})^{∗} = (cl K^{∘f})^{∗} = (K^{∘f})^{∗}. By Definition 3.4 of the dual cone, we have then

(K^{f})^{∗} = {v^{∗} ∈ ℝ^{n} × ℝ × ℝ | v^{T} v^{∗} ≥ 0 for all v ∈ K^{∘f}} ,  (7.1)

which translates into

(x^{∗}, θ^{∗}, κ^{∗}) ∈ (K^{f})^{∗} ⇔ x^{T} x^{∗} + θθ^{∗} + κκ^{∗} ≥ 0 for all (x, θ, κ) ∈ K^{∘f} .

Let us suppose first that κ^{∗} > 0. We find that

x^{T} x^{∗} + θθ^{∗} + κκ^{∗} ≥ 0 ∀(x, θ, κ) ∈ K^{∘f} ⇔ \frac{x^{T}}{θ} \frac{x^{∗}}{κ^{∗}} + \frac{θ^{∗}}{κ^{∗}} + \frac{κ}{θ} ≥ 0 ∀(x, θ, κ) ∈ K^{∘f} ,

which, since θ > 0 and κ is only restricted to its lower bound in the definition of K^{∘f}, is equivalent to

\frac{x^{T}}{θ} \frac{x^{∗}}{κ^{∗}} + \frac{θ^{∗}}{κ^{∗}} + \frac{θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ})}{θ} ≥ 0 ⇔ \frac{θ^{∗}}{κ^{∗}} ≥ −\frac{x^{T}}{θ} \frac{x^{∗}}{κ^{∗}} − ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ}) ∀(x, θ) s.t. \frac{x_{i}}{θ} ∈ dom f_{i} ,

where we could replace condition (x, θ, κ) ∈ K^{∘f} with the simpler requirement that x_{i}/θ belongs to the domain of f_{i} for all 1 ≤ i ≤ n. The key insight to have here is to note that the maximum of the right-hand side for all valid x and θ can be expressed with the conjugate functions f_{i}^{∗}, since

f_{i}^{∗}(−\frac{x_{i}^{∗}}{κ^{∗}}) = sup_{y∈ℝ} \{−y \frac{x_{i}^{∗}}{κ^{∗}} − f_{i}(y)\} = sup_{y∈dom f_{i}} \{−y \frac{x_{i}^{∗}}{κ^{∗}} − f_{i}(y)\} = sup_{(x_{i}/θ)∈dom f_{i}} \{−\frac{x_{i}}{θ} \frac{x_{i}^{∗}}{κ^{∗}} − f_{i}(\frac{x_{i}}{θ})\} .

Our condition is thus equivalent to

\frac{θ^{∗}}{κ^{∗}} ≥ ∑_{i=1}^{n} f_{i}^{∗}(−\frac{x_{i}^{∗}}{κ^{∗}}) ⇔ κ^{∗} ∑_{i=1}^{n} f_{i}^{∗}(−\frac{x_{i}^{∗}}{κ^{∗}}) ≤ θ^{∗} ,

which is exactly the same as saying that (−x^{∗}, κ^{∗}, θ^{∗}) ∈ K^{∘f^{∗}} or, using our definition of the switched cone, (x^{∗}, θ^{∗}, κ^{∗}) ∈ K_{s}^{∘f^{∗}}.

We have finally to examine the case κ^{∗} = 0, which will be done using an indirect approach. We have just shown that (K^{f})^{∗} ∩H = K_{s}^{∘f^{∗}}, where H is the open half-space defined by κ^{∗} > 0, i.e. H = ℝ^{n}×ℝ×ℝ_{++}. We are going to make use of Theorem 6.5 in [Roc70a], which essentially states that

cl(C_{1} ∩ C_{2}) = cl C_{1} ∩ cl C_{2} provided int C_{1} ∩ int C_{2} ≠ ∅ ,

i.e. that the closure of the intersection of two sets is the intersection of their closures, provided the intersection of their interiors is nonempty. We would like to apply this theorem to sets (K^{f})^{∗} and H. We first check that int(K^{f})^{∗} ∩ int H ≠ ∅. Indeed, we first have that int H = H. Moreover, it is easy to see that int K_{s}^{∘f^{∗}} ∩ H ≠ ∅ (see Theorem 7.2), which implies that int(K^{f})^{∗} ∩ H ≠ ∅ since K_{s}^{∘f^{∗}} ⊆ (K^{f})^{∗}. This allows us to apply the theorem and find that cl((K^{f})^{∗} ∩ H) = cl(K^{f})^{∗} ∩ cl H and, since (K^{f})^{∗} is closed, cl((K^{f})^{∗} ∩ H) = (K^{f})^{∗} ∩ cl H.

However, we cannot have a point with κ^{∗} < 0 in (K^{f})^{∗}. Indeed, choosing any point (x, θ, κ) in K^{∘f}, we have that (x, θ, κ′) ∈ K^{∘f} for all κ′ ≥ κ. If κ^{∗} < 0, we see that the quantity x^{T} x^{∗} + θθ^{∗} + κκ^{∗} can be made arbitrarily negative when κ′ → +∞, meaning that the point (x^{∗}, θ^{∗}, κ^{∗}) does not belong to our dual cone. Using the fact that cl H is the closed half-space defined by κ^{∗} ≥ 0 allows us to write that (K^{f})^{∗} ∩ cl H = (K^{f})^{∗}, which combined with the previous result shows that cl((K^{f})^{∗} ∩ H) = (K^{f})^{∗}.

Using finally the fact that (K^{f})^{∗} ∩ H = K_{s}^{∘f^{∗}}, we can conclude that (K^{f})^{∗} = cl K_{s}^{∘f^{∗}}, i.e. (K^{f})^{∗} = K_{s}^{f^{∗}}.

7.4 – An explicit definition of K^{f}  135

We note this proof is simpler than its counterpart for the L^{p} or the G_{2}^{n} cones, because the adequate use of K^{∘f} instead of K^{f} which allows an elegant treatment of the case κ^{∗} = 0.

The dual of a separable cone is thus equal, up to a change of sign and a permutation of two variables, to another separable cone based on conjugate functions.

Corollary 7.2. We also have (K_{s}^{f})^{∗} = K^{f^{∗}}, (K^{f^{∗}})^{∗} = K_{s}^{f} and (K_{s}^{f^{∗}})^{∗} = K^{f}.

Proof. Immediate considering on the one hand the symmetry between K^{f} and K_{s}^{f} and on the other hand the symmetry between f and f^{∗}.

Corollary 7.3. K^{f} and K_{s}^{f^{∗}} are solid and pointed.

Proof. We have already proved that K^{f} is solid which, for obvious symmetry reasons, implies that its switched counterpart K_{s}^{f^{∗}} is also solid. Since pointedness is the property that is dual to solidness (Theorem 3.3), noting that K^{f} = (K_{s}^{f^{∗}})^{∗} and K_{s}^{f^{∗}} = (K^{f})^{∗} is enough to prove that K^{f} and K_{s}^{f^{∗}} are also pointed.

7.4 An explicit definition of K^{f}

A drawback in our Definition 7.4 is the fact that it expresses K^{f} as the closure of another set, namely K^{∘f}. Since K^{∘f} ⊆ ℝ^{n}×ℝ_{++}×ℝ, we immediately have that K^{f} = cl K^{∘f} ⊆ ℝ^{n}×ℝ_{+}×ℝ, which shows that K^{f} can have points with a θ component equal to 0. This relates to the various conventions that had to be taken to handle the case of a zero denominator in the definitions of cones L^{p} and G_{2}^{n}.

The next theorem gives an explicit definition of K^{f}. It basically states that the points of K^{f} with a strictly positive θ are exactly the points of K^{∘f}, while the points with θ = 0 can be identified using the domain of the conjugate functions f_{i}^{∗}.

Theorem 7.4. We have

K^{f} = K^{∘f} ∪ {(x, 0, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ | x^{T} x^{∗} ≤ κ for all x_{i}^{∗} ∈ dom f_{i}^{∗}, 1 ≤ i ≤ n} .

Proof. A point (x, θ, κ) belongs to K^{f} if and only if there exists a sequence of points (x^{k}, θ^{k}, κ^{k}) belonging to K^{∘f} such that θ^{k} → θ, x^{k} → x and κ^{k} → κ. Let us suppose first that θ > 0.

It is obvious that points belonging to K^{∘f} satisfy θ > 0 and also belong to K^{f}. Let us show there are no other points in K^{f} with θ > 0. Using the fact that

θ^{k} ∑_{i=1}^{n} f_{i}(\frac{x_{i}^{k}}{θ^{k}} ) ≤ κ^{k} ,

we can take the limit and write

lim_{k→+∞} θ^{k} ∑_{i=1}^{n} f_{i}(\frac{x_{i}^{k}}{θ^{k}} ) ≤ lim_{k→+∞} κ^{k} = κ .  (7.2)

Using now the lower-semicontinuity of f_{i} we have that

lim_{k→+∞} θ^{k} ∑_{i=1}^{n} f_{i}(\frac{x_{i}^{k}}{θ^{k}} ) = θ ∑_{i=1}^{n} lim_{k→+∞} f_{i}(\frac{x_{i}^{k}}{θ^{k}} ) ≥ θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ} ) ,

since x_{i}^{k}/θ^{k} converges to x_{i}/θ, which shows eventually that

θ ∑_{i=1}^{n} f_{i}(\frac{x_{i}}{θ} ) ≤ κ ,

i.e. that (x, θ, κ) belongs to K^{∘f}. The sets K^{f} and K^{∘f} are thus identical when θ > 0.

Let us now examine the case θ = 0. Using Corollary 7.2, we have that K^{f} = (K_{s}^{f^{∗}})^{∗}. Looking now at equation (7.1) in the proof of Theorem 7.3, we see that points of K^{f} satisfying θ = 0 can be characterized by

(x, 0, κ) ∈ K^{f} ⇔ x^{T} x^{∗} + κκ^{∗} ≥ 0 for all (x^{∗}, θ^{∗}, κ^{∗}) ∈ K_{s}^{∘f^{∗}} ,

which is equivalent to

(x, 0, κ) ∈ K^{f} ⇔ x^{T} x^{∗} + κκ^{∗} ≥ 0 for all (−x^{∗}, κ^{∗}, θ^{∗}) ∈ K^{∘f^{∗}}
⇔ x^{T} (x^{∗}/κ^{∗}) + κ ≥ 0 for all (−x^{∗}, κ^{∗}, θ^{∗}) ∈ K^{∘f^{∗}} (using κ^{∗} > 0)
⇔ κ ≥ −x^{T} (x^{∗}/κ^{∗}) for all (−x^{∗}, κ^{∗}, θ^{∗}) ∈ K^{∘f^{∗}}
⇔ κ ≥ −x^{T} (x^{∗}/κ^{∗}) for all (−x_{i}^{∗}/κ^{∗}) ∈ dom f_{i}^{∗}, 1 ≤ i ≤ n
⇔ κ ≥ x^{T} x′^{∗} for all x′_{i}^{∗} ∈ dom f_{i}^{∗}, 1 ≤ i ≤ n (where x′^{∗} = x^{∗}/κ^{∗}) ,

which equivalent to the announced result.

7.5 Back to geometric and l_{p}-norm optimization

Let us check that our the separable cone K^{f} generalizes the cones L^{p} and G_{2}^{n} introduced in Chapters 4 and 6 for l_{p}-norm and geometric optimization. Special care will be taken to justify the conventions we had to introduce in order to handle the cases where θ = 0.

As mentioned in the introduction of this chapter, the L^{p} cone corresponds to the choice of f_{i} : x ↦ \frac{1}{p_{i}} |x|^{p_{i}}, which is easily seen to be a proper closed convex function. Let us compute the conjugate of this function: we have

f_{i}^{∗}(x^{∗}) : x^{∗} ↦ f_{i}^{∗}(x^{∗}) = sup_{x∈ℝ} \{xx^{∗} − \frac{|x|^{p_{i}}}{p_{i}} \} .

Introducing parameters q_{i} such that 1/p_{i} + 1/q_{i} = 1, we perform the maximization by letting the derivative of quantity appearing inside of the supremum equal to zero, which leads to x^{∗} = |x|^{p_{i}} /x and a supremum equal to

xx^{∗} − \frac{|x|^{p_{i}}}{p_{i}} = xx^{∗} − \frac{xx^{∗}}{p_{i}} = xx^{∗}(1 − \frac{1}{p_{i}} ) = \frac{xx^{∗}}{q_{i}} = \frac{|x|^{p_{i}}}{q_{i}} .

7.5 – Back to geometric and l_{p}-norm optimization  137

Using now

x^{∗} = |x|^{p_{i}} /x ⇒ |x^{∗}| = |x|^{p_{i}−1} ⇔ |x^{∗}|^{q_{i}} = |x|^{q_{i}(p_{i}−1)}

and

q_{i}(p_{i} − 1) = \frac{p_{i} − 1}{1 − 1/p_{i}} = (p_{i} − 1)\frac{p_{i}}{p_{i} − 1} = p_{i} ,

we find that |x^{∗}|^{q_{i}} = |x|^{p_{i}} and finally have that

f_{i}^{∗}(x^{∗}) = \frac{|x^{∗}|^{q_{i}}}{q_{i}} .

Let us check our convention when θ = 0. In light of Theorem 7.4, a point (x, 0, κ) will belong to K^{f} if and only if x^{T} x^{∗} ≤ κ for all x_{i}^{∗} ∈ dom f_{i}^{∗}, 1 ≤ i ≤ n. Since dom f_{i}^{∗} = ℝ, we see that this is possible if and only if x_{i} = 0, in which case we must have κ ≥ 0. This shows that

K^{f} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ | ∑_{i=1}^{n} \frac{1}{p_{i}} \frac{|x_{i}|^{p_{i}}}{θ^{p_{i}−1}} ≤ κ\} ,

with the convention

\frac{|x|}{0} = \begin{cases}+∞ & if x ≠ 0 , \\ 0 & if x = 0 , \end{cases}

which is exactly the definition of L^{p} given in Chapter 4 (one can also easily check that the dual L_{s}^{q} is equivalent to K_{s}^{f^{∗}}).

The geometric cone G_{2}^{n} is based on f_{i} : x ↦ e^{−x} but features a slight difference with our separable cone K^{f} since it requires x ≥ 0. However, the same effect can be obtained by restricting the effective domain of f_{i} to ℝ_{+}, i.e. letting

f_{i} : ℝ ↦ ℝ ∪ \{+∞\} : x ↦ \begin{cases}e^{−x} & when x ≥ 0 , \\ +∞ & when x < 0 . \end{cases}

It is straightforward to check that this function is convex proper and closed (note that the alternative choice f_{i}(0) = +∞ does not lead to a closed function). Its conjugate function can be computed in a straightforward manner, to find

f_{i}^{∗}(x^{∗}) : x^{∗} ↦ f_{i}^{∗}(x^{∗}) = \begin{cases}−1 & when x^{∗} ≤ −1 , \\ x^{∗} − x^{∗} log(−x^{∗}) & when − 1 < x^{∗} < 0 , \\ 0 & when x^{∗} = 0 , \\ +∞ & when 0 < x^{∗} . \end{cases}

According to Theorem 7.4, a point (x, 0, κ) will belong to G_{2}^{n} if and only if the product x^{T} x^{∗} is smaller than κ for all x_{i}^{∗} ∈ dom f_{i}^{∗}, 1 ≤ i ≤ n. Since dom f_{i}^{∗} = ℝ_{−}, we see that κ can only be finite when x ≥ 0, in which case it must satisfy κ ≥ 0. This justifies the convention e^{− \frac{x_{i}}{0}} = 0 that was made in Chapter 6, since it leads to

K^{f} = \{(x, θ, κ) ∈ ℝ_{+}^{n} × ℝ_{+} × ℝ_{+} | θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ κ\} ,

which is exactly the original definition of G_{2}^{n}. Let us compute its dual: we have

(K^{f})^{∗} = \{(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ^{n} × ℝ × ℝ_{+} | θ^{∗} ∑_{i=1}^{n} f_{i}^{∗}(−\frac{x_{i}^{∗}}{κ^{∗}}) ≤ θ^{∗}\} ,

which is equivalent to

\{(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ_{+}^{n} × ℝ × ℝ_{+} | θ^{∗} ≥ κ^{∗} ∑_{0<x_{i}^{∗}<κ^{∗}}(−\frac{x_{i}^{∗}}{κ^{∗}} + \frac{x_{i}^{∗}}{κ^{∗}} log \frac{x_{i}^{∗}}{κ^{∗}}) + κ^{∗} ∑_{x_{i}^{∗}≥κ^{∗}}(−1)\}

= \{(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ_{+}^{n} × ℝ × ℝ_{+} | θ^{∗} ≥ ∑_{0<x_{i}^{∗}<κ^{∗}}(x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} − x_{i}^{∗}) − ∑_{x_{i}^{∗}≥κ^{∗}} κ^{∗}\} ,

the original definition of the dual cone (G_{2}^{n})^{∗}. We first note that the effective domain of f_{i}^{∗} is responsible for restricting x^{∗} to ℝ_{+}^{n} and that we had to distinguish the cases −x_{i}^{∗}/κ^{∗} ≤ −1 and −x_{i}^{∗}/κ^{∗} > 1). Moreover, the special case κ^{∗} = 0 is handled correctly: we must have in that case −x^{T} x^{∗} ≤ θ^{∗} for all x ∈ dom f_{i}, which implies x^{∗} ≥ 0 and θ^{∗} ≥ 0, which is exactly what is expressed by our definition.

To conclude this section, we note that it is possible to give a simpler variant of our geometric cone G_{2}^{n}. Indeed, one can consider the negative exponential function on the whole real line, i.e. choose f_{i} : x ↦ e^{−x}, which is again closed, proper and convex. The expression of its conjugate function is simpler

f_{i}^{∗}(x^{∗}) : x^{∗} ↦ f_{i}^{∗}(x^{∗}) = \begin{cases}x^{∗} − x^{∗} log(−x^{∗}) & when x^{∗} < 0 , \\ 0 & when x^{∗} = 0 , \\ +∞ & when 0 < x^{∗} , \end{cases}

and leads to the following primal-dual pair of cones

K^{f} = \{(x, θ, κ) ∈ ℝ^{n} × ℝ_{+} × ℝ | θ ∑_{i=1}^{n} e^{− \frac{x_{i}}{θ}} ≤ κ\}
(K^{f})^{∗} = \{(x^{∗}, θ^{∗}, κ^{∗}) ∈ ℝ_{+}^{n} × ℝ × ℝ_{+} | θ^{∗} ≥ ∑_{x_{i}^{∗}>0}(x_{i}^{∗} log \frac{x_{i}^{∗}}{κ^{∗}} − x_{i}^{∗})\}

(note that negative components of x are now allowed in the primal, and that the distinction between x_{i}^{∗} < κ^{∗} and x_{i}^{∗} ≥ κ^{∗} has disappeared in the dual ; the convention e^{− \frac{x_{i}}{0}} = 0 stays valid when x_{i} ≥ 0 but has to be transformed to e^{− \frac{x_{i}}{0}} = +∞ for x_{i} < 0).

7.6 Separable convex optimization

The previous sections have introduced and studied the notion of separable cone, which encompasses the extended geometric cone G_{2}^{n} as well as the L^{p} cone used to model l_{p}-norm optimization. These separable cones are convex, closed, pointed and solid, and have a wellidentified dual, which makes them perfect candidates to be used in the framework of conic optimization described in Chapter 3.

We define now the class of separable convex optimization and show how its primal and dual problems can be modelled using the K^{f} and (K^{f})^{∗} cones.

As can be expected from the above developments, the structure of this class of problems is very similar to that of l_{p}-norm and geometric optimization. Indeed, we define two sets K = {1, 2, …, r}, I = {1, 2, …, n} and let {I_{k}}_{k∈K} be a partition of I into r classes. We also choose n closed, proper convex scalar functions f_{i} : ℝ ↦ ℝ ∪ {+∞}, whose conjugates will be denoted by f_{i}^{∗}. Finally, we assume that both int dom f_{i} and int dom f_{i}^{∗} are nonempty for all i ∈ I.

The data of our problems is given by two matrices A ∈ ℝ^{m×n} and F ∈ ℝ^{m×r} (whose columns will be denoted by a_{i}, i ∈ I and f_{k}, k ∈ K) and three column vectors b ∈ ℝ^{m}, c ∈ ℝ^{n} and d ∈ ℝ^{r}. The primal separable convex optimization problem consists in optimizing a linear function of a column vector y ∈ ℝ^{m} under a set of constraints involving functions f_{i} applied to linear forms, and can be written as

sup b^{T} y s.t. ∑_{i∈I_{k}} f_{i}(c_{i} − a_{i}^{T} y) ≤ d_{k} − f_{k}^{T} y ∀k ∈ K .  (SP)

Let us now model this problem with a conic formulation. We start by introducing an auxiliary vector of variables x^{∗} ∈ ℝ^{n} to represent the linear arguments of functions f_{i}, namely we let

x_{i}^{∗} = c_{i} − a_{i}^{T} y for all i ∈ I or, in matrix form, x^{∗} = c − A^{T} y ,

and we also introduce additional variables z^{∗} ∈ ℝ^{r} for the linear right-hand side of the inequalities

z_{k}^{∗} = d_{k} − f_{k}^{T} y for all k ∈ K or, in matrix form, z^{∗} = d − F^{T} y .

Our problem is now equivalent to

sup b^{T} y s.t. A^{T} y + x^{∗} = c, F^{T} y + z^{∗} = d and ∑_{i∈I_{k}} f_{i}(x_{i}^{∗}) ≤ z_{k}^{∗} ∀k ∈ K ,

where it is easy to plug our definition of the separable cone K^{f}, provided variables θ are fixed to one

sup b^{T} y s.t. A^{T} y + x^{∗} = c, F^{T} y + z^{∗} = d and (x_{I_{k}}^{∗}, 1, z_{k}^{∗}) ∈ K^{f^{k}} ∀k ∈ K

(where for convenience we defined f^{k} = (f_{i} | i ∈ I_{k}) for k ∈ K). We finally introduce an additional vector of fictitious variables v^{∗} ∈ ℝ^{r} whose components are fixed to one by additional linear constraints to find

sup b^{T} y s.t. A^{T} y + x^{∗} = c, F^{T} y + z^{∗} = d, v^{∗} = e and (x_{I_{k}}^{∗}, v_{k}^{∗}, z_{k}^{∗}) ∈ K^{f^{k}} ∀k ∈ K

(where e stands for the all-one vector). We point out that the description of the points belonging our separable cone when θ = 0 is not used here, since variables v_{r} cannot be equal to zero. Rewriting the linear constraints with a single matrix equality, we end up with

sup b^{T} y s.t. (\begin{matrix}A^{T} \\ F^{T} \\ 0 \end{matrix} ) y + (\begin{matrix}x^{∗} \\ z^{∗} \\ v^{∗} \end{matrix}) = (\begin{matrix}c \\ d \\ e \end{matrix}) and (x_{I_{k}}^{∗}, v_{k}^{∗}, z_{k}^{∗}) ∈ K^{f^{k}} ∀k ∈ K ,  (CSP)

which is exactly a conic optimization problem in the dual form (CD) of Chapter 3, using variables (ỹ, s̃), data (Ã, b̃, c̃) and a cone C^{∗} such that

ỹ = y, s̃ = (\begin{matrix}x^{∗} \\ z^{∗} \\ v^{∗} \end{matrix}) , Ã = (A F 0) , b̃ = b, c̃ = (\begin{matrix}c \\ d \\ e \end{matrix}) and C^{∗} = K^{f^{1}} × K^{f^{2}} × ⋯ × K^{f^{r}} ,

where C^{∗} has been defined according to Note 3.1, since we have to deal with multiple conic constraints involving disjoint sets of variables.

Using properties of K^{f} proved in the first part of this chapter, it is straightforward to show that C^{∗} is a solid, pointed, closed convex cone whose dual is

(C^{∗})^{∗} = C = K_{s}^{f^{1∗}} × K_{s}^{f^{2∗}} × ⋯ × K_{s}^{f^{r∗}} ,

another solid, pointed, closed convex cone (where we have defined f^{k∗} = (f^{i∗} | i ∈ I_{k}) for k ∈ K). This allows us to derive a dual problem to (CSP) in a completely mechanical way and find the following conic optimization problem, expressed in the primal form (CP) (since the dual of a problem in dual form is a problem in primal form):

inf (c^{T} d^{T} e^{T} ) (\begin{matrix}x \\ z \\ v \end{matrix}) s.t. (A F 0) (\begin{matrix}x \\ z \\ v \end{matrix}) = b and (x_{I_{k}}, v_{k}, z_{k}) ∈ K_{s}^{f^{k∗}} for all k ∈ K ,

which is equivalent to

inf c^{T} x + d^{T} z + e^{T} v s.t. Ax + F z = b and (x_{I_{k}}, v_{k}, z_{k}) ∈ K_{s}^{f^{k∗}} for all k ∈ K , (CSD)

where x ∈ ℝ^{n}, z ∈ ℝ^{r} and v ∈ ℝ^{r} are the dual variables we optimize. This problem can be simplified: developing the conic constraints, we find

inf c^{T} x + d^{T} z + e^{T} v s.t. \begin{cases}Ax + F z = b, z ≥ 0 \\ z_{k} ∑_{i∈I_{k}} f_{i}^{∗}(−\frac{x_{i}}{z_{k}} ) ≤ v_{k} & ∀k ∈ K | z_{k} > 0 \\ −x_{I_{k}}^{T}x_{I_{k}}^{∗} ≤ v_{k} ∀x_{I_{k}}^{∗} ∈ dom f_{I_{k}} & ∀k ∈ K | z_{k} = 0 \end{cases}

(where dom f_{I_{k}} is the cartesian product of all dom f_{i} such that i ∈ I_{k}), using the explicit definition of K^{f} given by Theorem 7.4. Finally, we can remove the v variables from the formulation since they are only lower bounded by the conic constraints, and have thus to attain this lower bound at any optimal solution. We can thus directly incorporate these terms into the objective function, which leads to the final dual separable optimization problem

inf ψ(x, z) = c^{T} x + d^{T} z + ∑_{k∈K|z_{k}>0} z_{k} ∑_{i∈I_{k}} f_{i}^{∗}(−\frac{x_{i}}{z_{k}} ) − ∑_{k∈K|z_{k}=0} inf_{x_{I_{k}}^{∗} ∈dom f_{I_{k}}} x_{I_{k}}^{T}x_{I_{k}}^{∗} (SD)
s.t. Ax + F z = b and z ≥ 0 .

Finally, we note that similarly to the case of geometric optimization, the special situation where F = 0 can lead to a further simplification of this dual problem. Indeed, since variables z_{k} do not appear in the linear constraints any more, they can be optimized separately and possibly be replaced in the objective function by a closed form of their optimal value.

7.7 Concluding remarks

In this chapter, we have generalized the cones G_{2}^{n} and L^{p} for geometric and l_{p}-norm optimization with the notion of separable cone K^{f}. This allowed us to present a new pair of primal-dual problems (SP)–(SD).

It is obvious that much more has to be said about this topic. We mention the following suggestions for further research:

⋄ Duality for the pair of primal-dual problems (SP)–(SD) can be studied using the theory presented in Chapter 3. Proving weak duality should be straightforward, as well as establishing the equivalent of the strong duality Theorem 3.5. Our feeling is that it should also be possible to prove that a zero duality gap can be guaranteed without any constraint qualification, because of the scalar nature of the functions used in the formulation.

⋄ Similarly to what was done in Chapter 4, it should be straightforward to build a selfconcordant barrier for the separable cone K^{f}, using as building blocks self-concordant barriers for the 2-dimensional epigraphs of functions f_{i}.

⋄ Finally, this formulation has the potential to model many more classes of convex problems. We mention the following three possibilities (see [Roc70a, p. 106])

– Let a ∈ ℝ_{++}. Functions of the type

f : x ↦ \begin{cases}−\sqrt{a^{2} − x^{2}} & if |x| ≤ a \\ +∞ & if |x| > a \end{cases} and f^{∗} : x^{∗} ↦ a\sqrt{1 + x^{∗2}}

are conjugate to each other, and could help modelling problems involving square roots or describing circles and ellipses.

– Let 0 < p < 1 and −∞ < q < 0 such that 1/p + 1/q = 1. Functions of the type

f : x ↦ \begin{cases}−\frac{1}{p}x^{p} & if x ≥ 0 \\ +∞ & if x < 0 \end{cases} and f^{∗} : x^{∗} ↦ \begin{cases}−\frac{1}{q}(−x^{∗})^{q} & if x^{∗} < 0 \\ +∞ & if x^{∗} ≥ 0 \end{cases}

are conjugate to each other, and appear to be able to model so-called CES functions [HvM97], which happen to be useful in production and consumer theory [Sat75].

– Functions

f : x ↦ \begin{cases}−\frac{1}{2} − log x & if x > 0 \\ +∞ & if x ≤ 0 \end{cases} and f^{∗} : x^{∗} ↦ \begin{cases}−\frac{1}{2} − log(−x^{∗}) & if x^{∗} < 0 \\ +∞ & if x^{∗} ≥ 0 \end{cases}

are conjugate to each other, and could be used in problems involving logarithms. They also feature the property that f^{∗}(x^{∗}) = f(−x^{∗}), which could add another level of symmetry between the corresponding primal K^{f} and dual K_{s}^{f^{∗}} cones.

We also point out that the definition of our separable convex optimization problems allows the use of different types of cones within the same constraint, which can lead for example to the formulation of a mixed geometric-l_{p}-norm optimization problem.

Part III

Approximations

Approximating geometric optimization with l_{p}-norm optimization

In this chapter, we demonstrate how to approximate geometric optimization with l_{p}-norm optimization. These two classes of problems are well known in structured convex optimization. We describe a family of l_{p}-norm optimization problems that can be made arbitrarily close to a geometric optimization problem, and show that the dual problems for these approximations are also approximating the dual geometric optimization problem. Finally, we use these approximations and the duality theory for l_{p}-norm optimization to derive simple proofs of the weak and strong duality theorems for geometric optimization.

8.1 Introduction

Let us recall first for convenience the formulation of the primal l_{p}-norm optimization problem (Pl_{p}) presented in chapter 4. Given two sets K = {1, 2, …, r} and I = {1, 2, …, n}, we let {I_{k}}_{k∈K} be a partition of I into r classes. The problem data is given by two matrices A ∈ ℝ^{m×n} and F ∈ ℝ^{m×r} (whose columns are be denoted by a_{i}, i ∈ I and f_{k}, k ∈ K) and four column vectors b ∈ ℝ^{m}, c ∈ ℝ^{n}, d ∈ ℝ^{r} and p ∈ ℝ^{n} such that p_{i} > 1 ∀i ∈ I. The primal l_{p}-norm optimization problem is

sup b^{T} y s.t. ∑_{i∈I_{k}} \frac{1}{p_{i}} |c_{i} − a_{i}^{T} y|^{p_{i}} ≤ d_{k} − f_{k}^{T} y ∀k ∈ K .  (Pl_{p})

146  8. Approximating geometric optimization with l_{p}-norm optimization

The purpose of this chapter is to show that this category of problems can be used to approximate another famous class of problems known as geometric optimization [DPZ67], presented in Chapter 5.

Using the same notations as above for sets K and I_{k}, k ∈ K, matrix A and vectors b, c and a_{i}, i ∈ I, we recall for convenience that the primal geometric optimization problem can be stated as

sup b^{T} y s.t. ∑_{i∈I_{k}} e^{a_{i}^{T} y−c_{i}} ≤ 1 ∀k ∈ K  (GP)

We will start by presenting in Section 8.2 an approximation of the exponential function, which is central in the definition of the constraints of a geometric optimization problem. This will allow us to present a family of l_{p}-norm optimization problems which can be made arbitrarily close to a primal geometric optimization problem. We derive in Section 8.3 a dual problem for this approximation, and show that the limiting case for these dual approximations is equivalent to the traditional dual geometric optimization problem. Using this family of pairs of primal-dual problems and the weak and strong duality theorems for l_{p}-norm optimization, we will then show how to derive the corresponding theorems for geometric optimization in a simple manner. Section 8.4 will conclude and present some topics for further research.

8.2 Approximating geometric optimization

In this section, we will show how geometric optimization problems can be approximated with l_{p}-norm optimization.

8.2.1 An approximation of the exponential function

A key ingredient in our approach is the function that will be used to approximate the exponential terms that arise within the constraints of (GP). Let α ∈ ℝ_{++} and let us define

g_{α} : ℝ_{+} ↦ ℝ_{+} : x ↦ |1 − \frac{x}{α}|^{α} .

We have the following lemma relating g_{α}(x) to e^{−x}:

Lemma 8.1. For any fixed x ∈ ℝ_{+}, we have that

g_{α}(x) ≤ e^{−x} ∀α ≥ x and e^{−x} < g_{α}(x) + α^{−1} ∀α > 0 ,  (8.1)

where the first inequality is tight if and only if x = 0. Moreover, we have

lim_{α→+∞} g_{α}(x) = e^{−x} .

Proof. Let us fix x ∈ ℝ_{+}. When 0 < α < x, we only have to prove the second inequality in (8.1), which is straightforward: we have e^{−x} < e^{−α} < α^{−1} < g_{α}(x) + α^{−1}, where we used the obvious inequalities e^{α} > α and g_{α}(x) > 0. Assuming α ≥ x for the rest of this proof, we define the auxiliary function h : ℝ_{++} ↦ ℝ : α ↦ log g_{α}(x). Using the Taylor expansion of log(1 − x) around x = 0

log(1 − x) = − ∑_{i=1}^{∞} \frac{x^{i}}{i} for all x such that |x| ≤ 1  (8.2)

we have

h(α) = α log|1 − \frac{x}{α}| = α log(1 − \frac{x}{α}) = − ∑_{i=1}^{∞} \frac{x^{i}}{iα^{i−1}} = −x − ∑_{i=2}^{∞} \frac{x^{i}}{iα^{i−1}}  (8.3)

(where we used the fact that \frac{x}{α} ≤ 1 to write the Taylor expansion). It is now clear that h(α) ≤ −x, with equality if and only if x = 0, which in turn implies that g_{α}(x) ≤ e^{−x}, with equality if and only if x = 0, which is the first inequality in (8.1).

The second inequality is equivalent, after multiplication by e^{x}, to

1 < e^{x}g_{α}(x) + e^{x}α^{−1} ⇔ 1 − e^{x}α^{−1} < e^{x}e^{h_{α}(x)} ⇔ 1 − e^{x}α^{−1} < e^{x+h_{α}(x)} .

This last inequality trivially holds when its left-hand side is negative, i.e. when α ≤ e^{x}. When α > e^{x}, we take the logarithm of both sides, use again the Taylor expansion (8.2) and the expression for h_{α}(x) in (8.3) to find

log(1 − e^{x}α^{−1}) < x + h_{α}(x) ⇔ − ∑_{i=1}^{∞} \frac{e^{x}^{i}}{iα^{i}} < − ∑_{i=2}^{∞} \frac{x^{i}}{iα^{i−1}} ⇔ 0 < ∑_{i=1}^{∞} \frac{1}{α^{i}} (\frac{e^{xi}}{i} − \frac{x^{i+1}}{i + 1}) .

This last inequality holds since each of the coefficients between parentheses can be shown to be strictly positive: writing the well-known inequality e^{a} > \frac{a^{n}}{n!} for a = xi and n = i + 1, we find

e^{xi} > \frac{(xi)^{i+1}}{(i + 1)!} ⇔ \frac{e^{xi}}{i} > \frac{x^{i+1}}{(i + 1)} \frac{i^{i}}{i!} ⇒ \frac{e^{xi}}{i} > \frac{x^{i+1}}{i + 1} ⇔ \frac{e^{xi}}{i} − \frac{x^{i+1}}{i + 1} > 0

(where we used i^{i} ≥ i! to derive the third inequality).

To conclude this proof, we note that (8.3) implies that lim_{α→+∞} h(α) = −x, which gives lim_{α→+∞} g_{α}(x) = e^{−x}, as announced. This last property can also be easily derived from the two inequalities in (8.1).

The first inequality in (8.1) and the limit of g_{α}(x) are well-known, and are sometimes used as definition for the real exponential function, while the second inequality in (8.1) is much less common.

8.2.2 An approximation using l_{p}-norm optimization

The formulation of the primal geometric optimization problem (GP) relies heavily on the exponential function. Since Lemma 8.1 shows that it is possible to approximate e^{−x} with increasing accuracy using the function g_{α}, we can consider using this function to formulate an approximation of problem (GP). The key observation we make here is that this approximation can be expressed as an l_{p}-norm optimization problem.

Indeed, let us fix α ∈ ℝ_{++} and write the approximate problem

sup b^{T} y s.t. ∑_{i∈I_{k}}(g_{α}(c_{i} − a_{i}^{T} y) + α^{−1}) ≤ 1 ∀k ∈ K .  (GP_{α})

We note that this problem is a restriction of the original problem (GP), i.e. that any y that is feasible for (GP_{α}) is also feasible for (GP), with the same objective value. This is indeed a direct consequence of the second inequality in (8.1), which implies for any y feasible for (GP_{α})

∑_{i∈I_{k}} e^{c_{i}−a_{i}^{T} y} < ∑_{i∈I_{k}}(g_{α}(c_{i} − a_{i}^{T} y) + α^{−1}) ≤ 1 .

We need now to transform the expressions g_{α}(c_{i} − a_{i}^{T} y) + α^{−1} to fit the format of the constraints of an l_{p}-norm optimization problem. Assuming that α > 1 for the rest of this chapter, we write

∑_{i∈I_{k}}(g_{α}(c_{i} − a_{i}^{T} y) + α^{−1}) ≤ 1 ⇔ ∑_{i∈I_{k}} g_{α}(c_{i} − a_{i}^{T} y) ≤ 1 − n_{k}α^{−1}

⇔ ∑_{i∈I_{k}} |1 − \frac{c_{i} − a_{i}^{T} y}{α} |^{α} ≤ 1 − n_{k}α^{−1}
⇔ ∑_{i∈I_{k}} |α − c_{i} + a_{i}^{T} y|^{α} ≤ α^{α}(1 − n_{k}α^{−1})

⇔ ∑_{i∈I_{k}} \frac{1}{α} |c_{i} − α − a_{i}^{T} y|^{α} ≤ α^{α−1}(1 − n_{k}α^{−1})

(where n_{k} is the number of elements in I_{k}), which allows us to write (GP_{α}) as

sup b^{T} y s.t. ∑_{i∈I_{k}} \frac{1}{α} |c_{i} − α − a_{i}^{T} y|^{α} ≤ α^{α−1}(1 − n_{k}α^{−1}) ∀k ∈ K .  (GP′_{α})

This is indeed an l_{p}-norm optimization in the form (Pl_{p}): dimensions m, n and r are the same in both problems, sets I, K and I_{k} are identical, the vector of exponents p satisfies p_{i} = α > 1 for all i ∈ I, matrix A and vector b are the same for both problems while matrix F is equal to zero. The only difference consists in vectors c̃ and d, which satisfy c̃_{i} = c_{i} − α and d_{k} = α^{α−1}(1 − n_{k}α^{−1}).

We have thus shown how to approximate a geometric optimization problem with a standard l_{p}-norm optimization problem. Solving this problem for a fixed value of α will give a feasible solution to the original geometric optimization problem. Letting α tend to +∞, the approximations g_{α}(c_{i} − a_{i}^{T} y) will be more and more accurate, and the corresponding feasible regions will approximate the feasible region of (GP) better and better. We can thus expect the optimal solutions of problems (GP′_{α}) to tend to an optimal solution of (GP). Indeed, this is the most common situation, but it does not happen in all the cases, as will be showed in the next section.

8.3 Deriving duality properties

The purpose of this section is to study the duality properties of our geometric optimization problem and its approximations. Namely, using the duality properties of l_{p}-norm optimization problems, we will derive the corresponding properties for geometric optimization, using our family of approximate problems.

8.3.1 Duality for l_{p}-norm optimization

Defining a vector q ∈ ℝ^{n} such that \frac{1}{p_{i}} + \frac{1}{q_{i}} = 1 for all i ∈ I, we recall from Chapter 4 that the dual problem for (Pl_{p}) consists in finding two vectors x ∈ ℝ^{n} and z ∈ ℝ^{r} that maximize a highly nonlinear objective while satisfying some linear equalities and nonnegativity constraints:

inf ψ(x, z) = c^{T} x + d^{T} z + ∑_{k∈K|z_{k}>0} z_{k} ∑_{i∈I_{k}} \frac{1}{q_{i}} |\frac{x_{i}}{z_{k}} |^{q_{i}} s.t. \{ \begin{matrix}Ax + F z = b and z ≥ 0 , \\ z_{k} = 0 ⇒ x_{i} = 0 ∀i ∈ I_{k} . \end{matrix} (Dl_{p})

Let us recall here for convenience from Chapter 4 the following duality properties for the pair of problems (Pl_{p})–(Dl_{p}):

Theorem 8.1 (Weak duality). If y is feasible for (Pl_{p}) and (x, z) is feasible for (Dl_{p}), we have ψ(x, z) ≥ b^{T} y.

Theorem 8.2 (Strong duality). If both problems (Pl_{p}) and (Dl_{p}) are feasible, the primal optimal objective value is attained with a zero duality gap, i.e.

p^{∗} = max b^{T} y s.t. ∑_{i∈I_{k}} \frac{1}{p_{i}} |c_{i} − a_{i}^{T} y|^{p_{i}} ≤ d_{k} − f_{k}^{T} y ∀k ∈ K

= inf ψ(x, z) s.t. \{ \begin{matrix}Ax + F z = b and z ≥ 0 \\ z_{k} = 0 ⇒ x_{i} = 0 ∀i ∈ I_{k} \end{matrix} = d^{∗} .

We would like to bring the reader’s attention to an interesting special case of dual l_{p}-norm optimization problem. When matrix F is identically equal to 0, i.e. when there are no pure linear terms in the constraints, and when all exponents p_{i} corresponding to same set I_{k} are equal to each other, i.e. when we have p_{i} = p^{k} ∀i ∈ I_{k} for all k ∈ K, problem (Dl_{p}) becomes

inf ψ(x, z) = c^{T} x + d^{T} z + ∑_{k∈K|z_{k}>0} \frac{z_{k}^{1−q^{k}}}{q^{k}} ∑_{i∈I_{k}} |x_{i}|^{q^{k}} s.t. \{ \begin{matrix}Ax = b and z ≥ 0 , \\ z_{k} = 0 ⇒ x_{i} = 0 ∀i ∈ I_{k} . \end{matrix} (Dl′_{p})

This kind of formulation arises in problems of approximation in l_{p}-norm, see [NN94, Section 6.3.2] and [Ter85, Section 11, page 98].

Since variables z_{k} do not appear any more in the linear constraints but only in the objective function ψ(x, z), we may try to find a closed form for their optimal value. Looking at one variable z_{k} at a time and isolating the corresponding terms in the objective, one finds d_{k}z_{k} + \frac{1}{q^{k}} z_{k}^{1−q^{k}} ∑_{i∈I_{k}} |x_{i}|^{q^{k}}, whose derivative is equal to d_{k} + \frac{1−q^{k}}{q^{k}} z_{k}^{−q^{k}} ∑_{i∈I_{k}} |x_{i}|^{q^{k}}. One easily sees that this quantity admits a single maximum when

z_{k} = (p^{k} d_{k})^{− \frac{1}{q^{k}}} ∥x_{I_{k}}∥_{q^{k}}

(where ∥·∥_{p} corresponds to the usual p-norm defined by ∥x∥_{p} = (∑_{i} |x_{i}|^{p})^{\frac{1}{p}} and x_{I_{k}} denotes the vector made of the components of x whose indices belong to I_{k}), which always satisfies the nonnegativity constraint in (Dl′_{p}) and gives after some straightforward computations a value of

d_{k}z_{k} + \frac{1}{q^{k}} z_{k}^{1−q^{k}} ∑_{i∈I_{k}} |x_{i}|^{q^{k}} = … = (1 + \frac{p^{k}}{q^{k}} )d_{k}z_{k} = p^{k} d_{k}z_{k} = (p^{k} d_{k})^{\frac{1}{p^{k}}} ∥x_{I_{k}}∥_{q^{k}}

for the two corresponding terms in the objective. Our dual problem (Dl′_{p}) becomes then

inf ψ(x) = c^{T} x + ∑_{k∈K}(p^{k} d_{k})^{\frac{1}{p^{k}}} ∥x_{I_{k}}∥_{q^{k}} s.t. Ax = b ,  (Dl′′_{p})

a great simplification when compared to (Dl′_{p}). One can check that the special treatment for the case z_{k} = 0 is well handled: indeed, z_{k} = 0 happens when x_{I_{k}} = 0, and the implication that is stated in the constraints of (Dl′_{p}) is thus satisfied.

It is interesting point out that problem(Dl′′_{p}) is essentially unconstrained, since it is wellknown that linear equalities can be removed from an optimization problem that does not feature other types constraints (assuming matrix A has rank l, one can for example use these equalities to express l variables as linear combinations of the other variables and pivot these l variables out of the formulation). We also observe that in this case a primal problem with p-norms leads to a dual problem with q-norms, a situation which is examined by Dax and Sreedharan in [DS97].

8.3.2 A dual for the approximate problem

We are now going to write the dual for the approximate problem (GP′_{α}). Since we are in the case where F = 0 and all p_{i}’s are equal to α, we can use the simplified version of the dual problem (Dl′′_{p}) and write

inf ψ_{α}(x) = c^{T} x − αe_{n}^{T} x + ∑_{k∈K} (α α^{α−1}(1 − n_{k}α^{−1}))^{\frac{1}{α}} ∥x_{I_{k}}∥_{β} s.t. Ax = b

(where e_{n} is a notation for the all-one n-dimensional column vector and β > 1 is a constant such that \frac{1}{α} + \frac{1}{β} = 1), which can be simplified to give

inf ψ_{α}(x) = c^{T} x − αe_{n}^{T} x + α ∑_{k∈K}(1 − n_{k}α^{−1})^{\frac{1}{α}} ∥x_{I_{k}}∥_{β} s.t. Ax = b .  (GD_{α})

We observe that the constraints and thus the feasible region of this problem are independent from α, which only appears in the objective function ψ_{α}(x). Intuitively, since problems (GP′_{α}) become closer and closer to (GP) as α tends to +∞, the corresponding dual problems (GP′_{α}) should approximate the dual of (GP) better and better. It is thus interesting to write down the limiting case for these problems, i.e. find the limit of ψ_{α} when α → +∞. Looking first at the terms that are related to single set of indices I_{k}, we write

ψ_{k,α}(x) = c_{I_{k}}^{T}x_{I_{k}} − αe_{n_{k}}^{T}x_{I_{k}} + α(1 − n_{k}α^{−1})^{\frac{1}{α}} ∥x_{I_{k}}∥_{β}
= c_{I_{k}}^{T}x_{I_{k}} − αe_{n_{k}}^{T}x_{I_{k}} + α ∥x_{I_{k}}∥_{1} − α ∥x_{I_{k}}∥_{1} + α(1 − n_{k}α^{−1})^{\frac{1}{α}} ∥x_{I_{k}}∥_{β}
= c_{I_{k}}^{T}x_{I_{k}} + α [∥x_{I_{k}}∥_{1} − e_{n_{k}}^{T}x_{I_{k}}] + α [(1 − n_{k}α^{−1})^{\frac{1}{α}} ∥x_{I_{k}}∥_{β} − ∥x_{I_{k}}∥_{1}]
= c_{I_{k}}^{T}x_{I_{k}} + α [∥x_{I_{k}}∥_{1} − e_{n_{k}}^{T}x_{I_{k}}] + \frac{β}{β − 1} [(1 − n_{k}α^{−1})^{\frac{1}{α}} ∥x_{I_{k}}∥_{β} − ∥x_{I_{k}}∥_{1}]

(where we used at the last line the fact that α = \frac{β}{β−1}). When α tends to +∞ (and thus β → 1), we have that the limit of ψ_{k,α}(x) is equal to

c_{I_{k}}^{T}x_{I_{k}} + lim_{α→+∞} α [∥x_{I_{k}}∥_{1} − e_{n_{k}}^{T}x_{I_{k}}] + lim_{\begin{matrix}α→+∞ \\ β→1 \end{matrix}} \frac{β}{β − 1} [(1 − n_{k}α^{−1})^{\frac{1}{α}} ∥x_{I_{k}}∥_{β} − ∥x_{I_{k}}∥_{1}]

= c_{I_{k}}^{T}x_{I_{k}} + lim_{α→+∞} α [∥x_{I_{k}}∥_{1} − e_{n_{k}}^{T}x_{I_{k}}] + lim_{β→1} \frac{∥x_{I_{k}}∥_{β} − ∥x_{I_{k}}∥_{1}}{β − 1}

The last term in this limit is equal to the derivative of the real function m_{k} : β ↦ ∥x_{I_{k}}∥_{β} at the point β = 1. We can check with some straightforward but lengthy computations that

m′_{k}(β) = \frac{∥x_{I_{k}}∥_{β}^{\frac{1}{β} −1}}{β^{2}} [β ∑_{i∈I_{k}|x_{i}>0} |x_{i}|^{β} log |x_{i}| − ∥x_{I_{k}}∥_{1} log ∥x_{I_{k}}∥_{1}] ,

which gives for β = 1

m′_{k}(1) = ∑_{i∈I_{k}|x_{i}>0} |x_{i}| log \frac{|x_{i}|}{∥x_{I_{k}}∥_{1}} ,

and leads to

lim_{\begin{matrix}α→+∞ \\ β→1 \end{matrix}} ψ_{k,α}(x) = c_{I_{k}}^{T}x_{I_{k}} + lim_{α→+∞} α [∥x_{I_{k}}∥_{1} − e_{n_{k}}^{T}x_{I_{k}}] + ∑_{i∈I_{k}|x_{i}>0} |x_{i}| log \frac{|x_{i}|}{∥x_{I_{k}}∥_{1}} .

It is easy to see that ∥x_{I_{k}}∥_{1} − e_{n_{k}}^{T}x_{I_{k}} ≥ 0, with equality if and only if x_{I_{k}} ≥ 0. This means that the limit of our objective ψ_{k,α}(x) will be +∞ unless x_{I_{k}} ≥ 0. An objective equal to +∞ for a minimization problem can be assimilated to an unfeasible problem, which means that the limit of our dual approximations (GD_{α}) admits the hidden constraint x_{I_{k}} ≥ 0. Gathering now all terms in the objective, we eventually find the limit of problems (GD_{α}) when α → +∞ to be

inf ϕ(x) = c^{T} x + ∑_{k∈K} ∑_{i∈I_{k}|x_{i}>0} x_{i} log \frac{x_{i}}{∑_{i∈I_{k}} x_{i}} s.t. Ax = b and x ≥ 0 ,  (GD)

which is exactly the dual geometric optimization problem that was presented in Chapter 5.

8.3.3 Duality for geometric optimization

Before we start to prove duality results for geometric optimization, we make a technical assumption on problem (GP), whose purpose will become clear further in this section: we assume that n_{k} ≥ 2 for all k ∈ K, i.e. forbid problems where a constraint is defined with a single exponential term. This can be done without any loss of generality, since a constraint of the form e^{a_{i}^{T} y−c_{i}} ≤ 1 can be equivalently rewritten as e^{a_{i}^{T} y−c_{i}−log 2} + e^{a_{i}^{T} y−c_{i}−log 2} ≤ 1.

Let us now prove the weak duality Theorem 5.9 for geometric optimization:

Theorem 8.3 (Weak duality). If y is feasible for (GP) and x is feasible for (GD), we have ϕ(x) ≥ b^{T} y.

Proof. Our objective is to prove this theorem using our family of primal-dual approximate problems (GP′_{α})–(GD_{α}). We first note that x is feasible for (GD_{α}) for every α, since the only constraints for this family of problems are the linear constraints Ax = b, which are also present in (GD). The situation is a little different on the primal side: the first inequality in (8.1) and feasibility of y for (GP) imply

∑_{i∈I_{k}} g_{α}(c_{i} − a_{i}^{T} y) ≤ ∑_{i∈I_{k}} e^{a_{i}^{T} y−c_{i}} ≤ 1 ,

with equality if and only if c_{i} −a_{i}^{T} y = 0 for all i ∈ I_{k}. But this cannot happen, since we would have ∑_{i∈I_{k}} e^{a_{i}^{T} y−c_{i}} = ∑_{i∈I_{k}} 1 = n_{k} > 1, because of our assumption on n_{k}, which contradicts the feasibility of y. We can conclude that the following strict inequality holds for all k ∈ K:

∑_{i∈I_{k}} g_{α}(c_{i} − a_{i}^{T} y) < 1 .

Since the set K is finite, this means that there exists a constant M such that for all α ≥ M,

∑_{i∈I_{k}} g_{α}(c_{i} − a_{i}^{T} y) ≤ 1 − n_{k}α^{−1} ∀k ∈ K ,

which in turn implies feasibility of y for problems (GP′_{α}) as soon as α ≥ M. Feasibility of both y and x for their respective problem allows us to apply the weak duality Theorem 8.1 of l_{p}-norm optimization to our pair of approximate problems (GP′_{α})–(GD_{α}), which implies ψ_{α}(x) ≥ b^{T} y for all α ≥ M. Taking now the limit of ψ_{α}(x) for α tending to +∞, which is finite and equal to ϕ(x) since x ≥ 0, we find that ϕ(x) ≥ b^{T} y, which is the announced inequality.

The strong duality Theorem 5.13 for geometric optimization is stated below. We note that contrary to the class of l_{p}-norm optimization problems, attainment cannot be guaranteed for any of the primal and dual optimum objective values.

Theorem 8.4. If both problems (GP) and (GD) are feasible, their optimum objective values p^{∗} and d^{∗}are equal.

Proof. As shown in the proof of the previous theorem, the existence of a feasible solution for (GP) and (GD) implies that problems (GP′_{α}) and (GD_{α}) are both feasible for all α greater than some constant M. Denoting by p_{α}^{∗} (resp. d_{α}^{∗}) the optimal objective value of problem (GP′_{α}) (resp. (GD_{α})), we can thus apply the strong duality Theorem 8.2 of l_{p}-norm optimization to these pairs of problems to find that p_{α}^{∗} = d_{α}^{∗} for all α ≥ M. Since all the dual approximate problems p_{α}^{∗} = d_{α}^{∗} share the same feasible region, it is clear that the optimal value corresponding to the limit of the objective ψ_{α} when α → +∞ is equal to the limit of the optimal objective values d_{α}^{∗} for α → +∞. Since the problem featuring this limiting objective has been shown to be equivalent to (GD) in Section 8.3.2 (including the hidden constraint x ≥ 0), we must have d^{∗} = lim_{α→+∞} d_{α}^{∗}. On the other hand, Theorem 8.2 guarantees for each of the problems (GP′_{α}) the existence of an optimal solution y_{α} that satisfies b^{T} y_{α} = p_{α}^{∗}. Since each of these solutions is also a feasible solution for (GP) (since problems (GP′_{α}) are restrictions of (GP)), which shares the same objective function, we have that the optimal objective value of (GP) p^{∗} is at least equal to b^{T} y_{α} for all α ≥ M, which implies p^{∗} ≥ lim_{α→+∞} b^{T} y_{α} = lim_{α→+∞} p_{α}^{∗} = lim_{α→+∞} d_{α}^{∗} = d^{∗}. Combining this last inequality with the easy consequence of the weak duality Theorem 8.3 that states d^{∗} ≥ p^{∗}, we end up with the announced equality p^{∗} = d^{∗}.

The reason why attainment of the primal optimum objective value cannot be guaranteed is that the sequence y_{α} may not have a finite limit point, a justification that is very similar to the one that was given in the concluding remarks of Chapter 5.

8.4 Concluding remarks

In this chapter, we have shown that the important class of geometric optimization problems can be approximated with l_{p}-norm optimization.

We have indeed described a parameterized family of primal and dual l_{p}-norm optimization problems, which can be made arbitrarily close to the geometric primal and dual problems. It is worth to note that the primal approximations are restrictions of the original geometric primal problem, sharing the same objective function, while the dual approximations share essentially the same constraints as the original geometric dual problem (except for the nonnegativity constraints) but feature a different objective.

Another possible approach would be to work with relaxations instead of restrictions on the primal side, using the first inequality in (8.1) instead of the second one, leading to the following problem:

sup b^{T} y s.t. ∑_{i∈I_{k}} g_{α}(c_{i} − a_{i}^{T} y) ≤ 1 ∀k ∈ K .

However, two problems arise in this setting:

⋄ the first inequality in (8.1) is only valid when α ≥ x, which means we would have to add a set of explicit linear inequalities c_{i} −a_{i}^{T} y ≤ α to our approximations, which would make them and their dual problems more difficult to handle, ⋄ following the same line of reasoning as in the proof of Theorem 8.2, we would end up with

another family of optimal solutions y_{α} for the approximate problems; however, since all of these problems are relaxations, we would have no guarantee that any of the optimal vectors y_{α} are feasible for the original primal geometric optimization problem, which would prevent us to conclude that the duality gap is equal to zero. This would only show that there is a family of asymptotically feasible primal solutions with their objective values tending to the objective value of the dual, a fact that is always true in convex optimization (this is indeed the essence of the alternate strong duality Theorem 3.6, related to the notion of subvalue, see Chapter 3).

To conclude, we note that our approximate problems belong to a very special subcategory of l_{p}-norm optimization problem, since they satisfy F = 0. It might be fruitful to investigate which class of generalized geometric optimization problems can be approximated with general l_{p}-norm optimization problems, a topic we leave for further research.

Computational experiments with a linear approximation

of second-order cone optimization

In this chapter, we present and improve a polyhedral approximation of the second-order cone due to Ben-Tal and Nemirovski [BTN98]. We also discuss several ways of reducing the size of this approximation. This construction allows us to approximate second-order cone optimization problems with linear optimization.

We implement this scheme and conduct computational experiments dealing with two classes of second-order cone problems: the first one involves trusstopology design and uses a large number of second-order cones with relatively small dimensions, while the second one models convex quadratic optimization problems with a single large second-order cone.

9.1 Introduction

Chapter 3 deals with conic optimization, which is a powerful setting that relies on convex cones to formulate convex problems. We recall here the standard conic primal-dual pair for convenience

inf_{x∈ℝ^{n}} c^{T} x s.t. Ax = b and x ∈ C  (CP)
sup_{y∈ℝ^{m},x^{∗}∈ℝ^{n}} b^{T} y s.t. A^{T} y + x^{∗} = c and x^{∗} ∈ C^{∗} ,  (CD)

where x and (y, x^{∗}) are the primal and dual variables, A is a m × n matrix, b and c are m and n-dimensional column vectors, C ⊆ ℝ^{n} is a closed pointed solid convex cone and C^{∗} ⊆ ℝ^{n} is its dual cone, defined by C^{∗} = {x^{∗} ∈ ℝ^{n} | x^{T} x^{∗} ≥ 0 ∀x ∈ C}.

Different types of convex cones lead to different classes of problems: for example, linear optimization uses the nonnegative orthant ℝ_{+}^{n} while semidefinite optimization relies on the set of positive semidefinite matrices 𝕊_{+}^{n} (see Chapter 3). In this chapter, we will focus the second-order cone, also known as Lorentz cone or ice-cream cone, which leads to second-order cone optimization. It is defined as follows:

Definition 9.1. The second order cone 𝕃^{n} is the subset of ℝ^{n+1} defined by

𝕃^{n} = {(r, x) ∈ ℝ × ℝ^{n} | ∥x∥ ≤ r} ,

where ∥·∥ denotes the usual Euclidean norm on ℝ^{n}.

It is indeed straightforward to check that this is a closed pointed solid convex cone (it is in fact the epigraph of the Euclidean norm). Another interesting property of 𝕃^{n} is the fact that it is self-dual, i.e. (𝕃^{n})^{∗} = 𝕃^{n}.

The standard second-order cone problems are based on the cartesian product of several second-order cones, which can be formalized using r constants n_{k} ∈ ℕ, 1 ≤ k ≤ r such that ∑_{k=1}^{r}(n_{k} +1) = n and defining C = 𝕃^{n_{1}} ×𝕃^{n_{2}} ×⋯×𝕃^{n_{r}}. This set is obviously also a self-dual closed convex cone, which allows us to rewrite problems (CP) and (CD) as

inf c^{T} x s.t. Ax = b and x^{k} ∈ 𝕃^{n_{k}} ∀k = 1, 2, …, r  (9.1)
sup b^{T} y s.t. A^{T} y + x^{∗} = c and x^{k∗} ∈ 𝕃^{n_{k}} ∀k = 1, 2, …, r ,  (9.2)

where vectors x and x^{∗} have been split into r subvectors (x^{1}, x^{2}, …, x^{r}) and (x^{1∗}, x^{2∗}, …, x^{r∗}) with x^{k} ∈ ℝ^{n_{k}+1} and x^{k∗} ∈ ℝ^{n_{k}+1} for all k = 1, …, r. It is usually more practical to pivot out variables x^{∗} in the dual problem (9.2), i.e. write them as a function of vector y. Splitting matrix A into (A^{1}, A^{2}, …, A^{r}) with A^{k} ∈ ℝ^{m×(n_{k}+1)} and vector c into (c^{1}, c^{2}, …, c^{r}) with c^{k} ∈ ℝ^{n_{k}+1}, we have x^{k∗} = c^{k} − A^{k}^{T} y. The last step is to isolate the first column in A^{k} and the first component in c^{k}, i.e. letting A^{k} = (f^{k}, G^{k}) with f^{k} ∈ ℝ^{m} and G^{k} ∈ ℝ^{m×n_{k}} and c^{k} = (d^{k}, h^{k}) with d^{k} ∈ R and h^{k} ∈ ℝ^{n_{k}}, we can rewrite the dual problem (9.2) as

sup b^{T} y s.t. ∥G^{k}^{T} y + h^{k}∥ ≤ f^{k}^{T} y + d^{k} ∀k = 1, 2, …, r ,

which is more convenient to formulate real-world problems (we also note that these constraints bear a certain similarity to l_{p}-norm optimization constraints, see Chapter 4).

Second-order optimization admits many different well-known classes of optimization problems as special cases, such as linear optimization, linearly and quadratically constrained convex quadratic optimization, robust linear optimization, matrix-fractional problems and problems with hyperbolic constraints (see the survey [LVBL98]). Applications arise in various fields such as engineering (antenna array design, finite impulse response filter design, truss design) and finance (portfolio optimization), see again [LVBL98].

From the computational point of view, second-order cone optimization is a relatively young field if compared to linear and quadratic optimization (for example, the leading commercial linear and quadratic solvers do not yet offer the option of solving second-order cone optimization problems). This observation led Ben-Tal and Nemirovski to develop an interesting alternative approach to solving second-order cone problems: they show in [BTN98] that it is possible to write a polyhedral approximation of the second order cone 𝕃^{n} with a prescribed accuracy ϵ using a number of variables and constraints that is polynomial in n and log \frac{1}{ϵ}. This implies that second-order cone optimization problems can be approximated with an arbitrarily prescribed accuracy by linear optimization problems using this polyhedral approximation.

This potentially allows the approximate resolution of large-scale second-order cone problems using state of the art linear solvers, capable of handling problems with hundreds of thousands of variables and constraints.

This chapter is organized as follows: Section 9.2 presents a polyhedral approximation of the second-order cone. This construction relies on a decomposition scheme based on threedimensional second order cones. We present first an efficient way to approximate these cones and then show how to combine them in order to approximate a second-order cone of higher dimension, which ultimately gives a method to approximate any second-order cone optimization problem with a linear problem. Section 9.3 reports our computational experiments with this scheme. After a presentation of our implementation and some related issues, we describe two classes of second-order problems: truss-topology design problems and convex quadratic optimization problems. We present and discuss the results of our computational experiments, highlighting when necessary the particular features of each class of problems (guaranteed accuracy, alternative formulations). We conclude this chapter with a few remarks and suggestions for further research.

9.2 Approximating second-order cone optimization

In this section, we present a polyhedral approximation of the second-order cone 𝕃^{n} which allows us to derive a linearizing scheme for second-order cone optimization. It is a variation of the construction of Ben-Tal and Nemirovski that features slightly better properties.

9.2.1 Principle

The principle that lies behind their approximation is twofold:

a. Decomposition. Since the Lorentz cone 𝕃^{n} is a n + 1-dimensional subset, any circumscribed polyhedral cone around 𝕃^{n} is bound to have its number of facets growing exponentially with the dimension n, i.e. will need an exponential number of linear inequalities to be defined. The remedy is to decompose the second-order cone into a polynomial number of smaller second-order cones with fixed dimension, for which a good polyhedral approximation can be found. In the present case, 𝕃^{n} can be decomposed into n − 1 three-dimensional second-order cones 𝕃^{2}, at the price of introducing n − 2 additional variables (see Section 9.2.2).

b. Projection. Even the three-dimensional second-order cone 𝕃^{2} is not too easy to approximate: the most obvious way to proceed, a regular circumscribed polyhedral cone, requires hundreds of inequalities even for an approximation with modest accuracy (see Section 9.2.3). The key idea to lower the number of inequalities is to introduce several additional variables, i.e. lift the approximating polyhedron into a higher dimensional space and consider its projection onto a (n + 1)-dimensional subspace as the approximation of 𝕃^{n} (see Section 9.2.4).

To summarize, the introduction of a certain number of additional variables, combined with a projection, can be traded against a much lower number of inequality constraints defining the polyhedron. We first concentrate on the decomposition of 𝕃^{n} into smaller second-order cones.

9.2.2 Decomposition

Let us start with the following equivalent definition of 𝕃^{n}

𝕃^{n} = {(r, x_{1}, x_{2}, …, x_{n}) ∈ ℝ_{+} × ℝ^{n} | ∑_{i=1}^{n} x_{i}^{2} ≤ r^{2}} .

Introducing a vector of ⌊\frac{n}{2}⌋ additional variables y = (y_{1}, y_{2}, …, y_{⌊ \frac{n}{2} ⌋}), we consider the set 𝕃^{n}′ defined by

\{(r, x, y) ∈ ℝ_{+} × ℝ^{n+⌊ \frac{n}{2} ⌋} | x_{2i−1}^{2} + x_{2i}^{2} ≤ y_{i}^{2}, 1 ≤ i ≤ ⌊\frac{n}{2}⌋, \begin{cases}∑i=1^{\frac{n}{2}} y_{i}^{2} & ≤ r^{2} (n even) \\ ∑i=1^{\frac{n}{2}} y_{i}^{2} + x_{n}^{2} & ≤ r^{2} (n odd) \end{cases} \} .

It it straightforward to prove that the projection of this set on the subspace of its first n + 1 variables (r, x_{1}, …, x_{n}) is equal to 𝕃^{n}, i.e.

(r, x) ∈ 𝕃^{n} ⇔ ∃y ∈ ℝ^{⌊ \frac{n}{2} ⌋} s.t. (r, x, y) ∈ 𝕃^{n}′ .

It is also worth to point out that all the constraints defining 𝕃^{n}′ are second-order cone constraints, i.e. that 𝕃^{n}′ can also be written as

\{(r, x, y) ∈ ℝ_{+} × ℝ^{n+⌊ \frac{n}{2} ⌋} | (y_{i}, x_{2i−1}, x_{2i}) ∈ 𝕃^{2}, 1 ≤ i ≤ ⌊\frac{n}{2}⌋, \begin{cases}(r, y) & ∈ 𝕃^{⌈ \frac{n}{2} ⌉} (n even) \\ (r, y, x_{n}) & ∈ 𝕃^{⌈ \frac{n}{2} ⌉} (n odd) \end{cases}\} .

(9.3)

This means that 𝕃^{n} can be decomposed into ⌊\frac{n}{2}⌋ 3-dimensional second-order cones and a single 𝕃^{⌈ \frac{n}{2} ⌉} second-order cone, at the price of introducing ⌊\frac{n}{2}⌋ auxiliary variables. This procedure can be applied recursively to the largest of the remaining second-order cone 𝕃^{⌈ \frac{n}{2} ⌉} until it also becomes equal to 𝕃^{2}.

It is not too difficult to see that there are in the final expression n − 1 second-order cones 𝕃^{2} and n − 2 additional y_{i} variables. Indeed, the addition of each small cone 𝕃^{2} reduces the size of the largest cone by one, since we remove two variables from this cone (the last two variables in 𝕃^{2}) but replace them with a single new variable (the first variable in 𝕃^{2}).

Since we start with this largest cone equal 𝕃^{n} and stop when its size is equal to 2, we need n − 2 small cones along with n − 2 auxiliary variables to reduce the cone to 𝕃^{2}. But this last 𝕃^{2} cone also has to be counted, which gives then a total number of cones equal to n − 1.

The existence of this decomposition implies that any second-order cone optimization problem can be transformed into a problem using only 3-dimensional second-order cones, using the construction above. We note however that strictly speaking, the resulting formulation is not a conic problem, since some variables belong to two different cones at the same time. It is nonetheless possible to add a extra variable for each shared variable, along with a constraint to make them equal on the feasible region, to convert this formulation into the strict conic format (CP)–(CD).

9.2.3 A first approximation of 𝕃^{2}

The previous section has shown that we can focus our attention on approximations of the 3-dimensional second-order. Moreover, it seems reasonable to require this approximation to be a cone itself too. Taking into account this additional assumption, we can take advantage of the homogeneity property of these cones to write

(r, x_{1}, x_{2}) ∈ 𝕃^{2} ⇔ (1, \frac{x_{1}}{r} , \frac{x_{2}}{r}) ∈ 𝕃^{2} ,  (9.4)

which basically means we can fix r = 1 and look for a polyhedral approximation of the resulting set

{x ∈ ℝ^{2} | (1, x) ∈ 𝕃^{2}} = {x ∈ ℝ^{2} | x_{1}^{2} + x_{2}^{2} ≤ 1} = B_{2}(1) ,

which is exactly the disc of radius one in ℝ^{2}. Any approximating polyhedron for B_{2}(2) will be then later straightforwardly converted into a polyhedral cone approximating 𝕃^{2}, using the additional homogenizing variable r.

At this point, we have to introduce a measure of the quality of our approximations. A natural choice for this measure is to state that a polyhedron P ⊆ ℝ^{2} is a ϵ-approximation of B_{2}(1) if and only we have the double inclusion B_{2}(1) ⊆ P ⊆ B_{2}(1 + ϵ), i.e. the polyhedron contains the unit disc but lies entirely within the disc of radius 1 + ϵ.

The most obvious approximation of the unit disc is the regular m-polyhedron P_{m}, which is described by m linear inequalities. We have the following theorem:

Theorem 9.1. The regular polyhedron with m sides is an approximation of the unit disc B_{2}(1) with accuracy ϵ = cos(\frac{π}{m})^{−1} − 1.

Proof. The proof is quite straightforward: looking at Figure 9.1 (which represents the case m = 8), we see that angle ∠AOM is equal to \frac{π}{m} and thus that |OA| cos(\frac{π}{m}) = |OM| = 1. Our measure of quality is then equal to ϵ = |OA| − 1 = cos(\frac{π}{m})^{−1} − 1, as announced.

Figure 9.1: Approximating B_{2}(1) with a regular octagon.

This result is not very satisfying: since cos(x)^{−1} ≈ 1 + \frac{x^{2}}{2} when x is small, we have that ϵ ≈ \frac{π^{2}}{2m^{2}} when m is large, which means that doubling the number of inequalities only divides the accuracy by four. For example, approximating B_{2}(1) with the relatively modest accuracy 10^{−4} would already take a 223-sided polyhedron, i.e. more than 200 linear inequalities.

9.2.4 A better approximation of 𝕃^{2}

As outlined in Section 9.2.1, the key idea introduced by Ben-Tal and Nemirovski to obtain a better polyhedral approximation is to consider the projection of a polyhedron belonging to a higher dimensional space. The construction we are going to present here is a variation of the one described in [BTN98], featuring slightly better parameters and a more transparent proof.

Let us introduce an integer parameter k ≥ 2 and consider the set D_{k} ⊆ ℝ^{2k+2} defined as

\{(α_{0}, …, α_{k}, β_{0}, …, β_{k}) ∈ ℝ^{2k+2} | \{ \begin{matrix}α_{i+1} & = & α_{i} cos \frac{π}{2^{i}} & + & β_{i} sin \frac{π}{2^{i}} \\ β_{i+1} & ≥ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ −β_{i+1} & ≤ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ 1 & = & α_{k} cos \frac{π}{2^{k}} & + & β_{k} sin \frac{π}{2^{k}} \end{matrix} ∀ 0 ≤ i < k\} .

This set is obviously a polyhedron^{1}, since its defining constraints consist in k + 1 linear equalities and 2k inequalities. The following theorem gives some insight about the structure of this set.

1Strictly speaking, this set is not a full-dimensional polyhedron in ℝ^{2k+2} because of the additional linear constraints but this has no incidence on our purpose.

Theorem 9.2. The projection of the set D_{k} on the subspace of its two variables (α_{0}, β_{0}) is equal to the regular 2^{k}-sided polyhedron, i.e. we have

(α_{0}, β_{0}) ∈ P_{2^{k}} ⇔ ∃(α_{1}, …, α_{k}, β_{1}, …, β_{k}) ∈ ℝ^{2k} | (α_{0}, …, α_{k}, β_{0}, …, β_{k}) ∈ D_{k} .

Proof. To fix ideas, we are going to present some figures corresponding to the case k = 3, but our reasoning will of course be valid for all k ≥ 2. Looking at Figure 9.1, which depicts P_{2^{3}}, we see that the last equality in the definition of D_{k} describes the line AM. Indeed, we have A = (cos(\frac{π}{2^{k}})^{−1}, 0) and M = (cos(\frac{π}{2^{k}}), sin(\frac{π}{2^{k}})) and it is straightforward that both of these points satisfy the last equality in the definition of D_{k}.

Recall now that the application

R_{θ} : ℝ^{2} ↦ ℝ^{2} : (\begin{matrix}x \\ y \end{matrix}) ↦ R_{θ} (\begin{matrix}x \\ y \end{matrix}) = (\begin{matrix}x cos θ + y sin θ \\ −x sin θ + y cos θ \end{matrix})

is a clockwise rotation around the origin with angle θ. Calling P_{i} the point of ℝ^{2} whose coordinates are (α_{i}, β_{i}) and P̂_{i} = (α̂_{i}, β̂_{i}) the image of P_{i} by rotation R_{π/2^{i}}, we have that the first three constraints in the definition of D_{k} are equivalent to α_{i+1} = α̂_{i}, β_{i+1} ≥ β̂_{i} and −β_{i+1} ≤ β̂_{i}. These last two inequalities rewritten as −β_{i+1} ≤ β̂_{i} ≤ β_{i+1} immediately imply that β_{i+1} has to be nonnegative. Under this assumption, we call P̄_{i} the points whose coordinates are (α_{i}, −β_{i}) and find that these three constraints are equivalent to saying that P̂_{i} ∈ [P_{i+1} P̄_{i+1}]. In other words, the point P̂_{i} has to belong to a vertical segment [P_{i+1} P̄_{i+1}] such that P_{i+1} has its second coordinate nonnegative. Since P̂_{i} is the image of P_{i} by a rotation of angle π/2^{i}, saying that P̂_{i} belongs to some set is equivalent to saying that P_{i} belongs to the image of this set by the inverse rotation. In our case, this means in fine that P_{i} has to belong to the image by a rotation of angle −π/2^{i} of a segment [P_{i+1} P̄_{i+1}] such that P_{i+1} has its second coordinate nonnegative.

We can now specialize this result to i = k − 1. Recall that P_{k} is known to belong to the line AB. According to the above discussion, we have first to restrict this set to its points with a nonnegative β_{k}, which gives the half line [AB. Taking the union of all segments [P_{k} P̄_{k}] for all possible P_{k}’s gives the region bounded by half lines [AB and [AB′. Taking finally the image of this set by a rotation of angle −π/2^{k−1}, we find that P_{k−1} has to belong to the region bounded by half lines [BA and [BC.

We can now iterate this procedure and describe the set of points P_{k−2}, P_{k−3}, etc. Indeed, using exactly the same reasoning, we find that the set of points P_{i−1} can be deduced from the set of points P_{i} with a three-step procedure:

a. Restrict the set of points P_{i} to those with a nonnegative β_{i} coordinate.

b. Consider the union of segments [P_{i} P̄_{i}] where P_{i} belongs to the above restricted set, i.e. add for each point (α_{i}, β_{i}) the set of points (α_{i}, x) for all x ranging from −β_{i} to β_{i}.

c. Rotate this union counterclockwise around the origin with an angle equal to π/2^{i} to find the set of points P_{i−1}.

In the case of our example with k = 3, we have already shown that the set {P_{k}} = {P_{3}} = [AB and {P_{k−1}} = {P_{2}} is the region bounded by [BA ∪ [BC. Going on with the procedure described above, we readily find that {P_{k−2}} = {P_{1}} is the region bounded by the polygonal line [ABCDE] while {P_{k−3}} = {P_{0}} is the complete octagon ABCDED′C′B′A, which is the expected result (see Figure 9.2 for the corresponding pictures). It is not difficult to see that in the general case {P_{k−i}} is a set bounded by 2^{i} consecutive sides of P_{2^{k}}, which means we always end up with {P_{0}} equal to the whole regular 2^{k}-sided polyhedron P_{2^{k}}. This completes the proof since the set of points P_{0} is the projection of D_{k} on the subspace of the two variables (α_{0}, β_{0}).

Figure 9.2: The sets of points P_{3}, P_{2}, P_{1} and P_{0} when k = 3.

This theorem allows us to derive quite easily a polyhedral approximation for B_{2}(1).

Corollary 9.1. The projection of D_{k} on the subspace of its two variables (α_{0}, β_{0}) is a polyhedral approximation of B_{2}(1) with accuracy ϵ = cos(\frac{π}{2^{k}})^{−1} − 1.

Proof. Straightforward application of Theorems 9.1 and 9.2.

This approximation is much better than the previous one: we have here that ϵ ≈ \frac{π^{2}}{2^{2k+1}} , which means that dividing the accuracy by four can be achieved by increasing k by 1, which corresponds to adding 2 variables, 1 equality and 2 inequality constraints (compare to the previous situation which needed to double the number of inequalities to reach to same goal). For example, an accuracy of ϵ = 10^{−4} can be obtained with k = 8, i.e. with 16 inequalities, 9 equalities and 18 variables (as opposed to 223 inequalities with the previous approach).

We are now in position to convert this polyhedral approximation of B_{2}(1) into an approximation of 𝕃^{2}. We define the set L_{k} ∈ ℝ^{2k+3} as

\{(r, α_{0}, …, α_{k}, β_{0}, …, β_{k}) ∈ ℝ^{2k+3} | \{ \begin{matrix}α_{i+1} & = & α_{i} cos \frac{π}{2^{i}} & + & β_{i} sin \frac{π}{2^{i}} \\ β_{i+1} & ≥ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ −β_{i+1} & ≤ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ r & = & α_{k} cos \frac{π}{2^{k}} & + & β_{k} sin \frac{π}{2^{k}} \end{matrix} ∀ 0 ≤ i < k\} .

Note the close resemblance between this set and D_{k}, the only difference being the introduction of an additional variable r in the last equality constraint. This set L_{k} is our final polyhedral approximation of 𝕃^{2}. Obviously, before we give proof of this fact, we need a measure of the quality of an approximation in the case of a second-order cone. This is the purpose of the next definition.

Definition 9.2. A set S ⊆ ℝ^{n+1} is said to be an ϵ-approximation of the second-order cone 𝕃^{n} if and only if we have

𝕃^{n} ⊆ S ⊆ 𝕃_{ϵ}^{n} = {(r, x) ∈ ℝ × ℝ^{n} | ∥x∥ ≤ (1 + ϵ)r}

where 𝕃_{ϵ}^{n} is an ϵ-relaxed second-order cone.

This definition extends our definition of ϵ-approximation for the unit disc B_{2}(1). The next theorem demonstrates how Corollary 9.1 on the accuracy of the polyhedral approximation D_{k} for B_{2}(1) can be converted into a result on the accuracy of L_{k} for 𝕃^{2}.

Theorem 9.3. The projection of L_{k} on the subspace of its three variables (r, α_{0}, β_{0}) is a polyhedral approximation of 𝕃^{2} with accuracy ϵ = cos(\frac{π}{2^{k}})^{−1} − 1.

Proof. Assuming r > 0 for the moment, we first establish a link between D_{k} and L_{k}. It is indeed straightforward to check using the corresponding definitions that the following equivalence holds

(r, α_{0}, …, α_{k}, β_{0}, …, β_{k}) ∈ L_{k} ⇔ (\frac{α_{0}}{r} , …, \frac{α_{k}}{r} , \frac{β_{0}}{r} , …, \frac{β_{k}}{r}) ∈ D_{k} ,  (9.5)

since

\{ \begin{matrix}α_{i+1} & = & α_{i} cos \frac{π}{2^{i}} & + & β_{i} sin \frac{π}{2^{i}} \\ β_{i+1} & ≥ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ −β_{i+1} & ≤ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ r & = & α_{k} cos \frac{π}{2^{k}} & + & β_{k} sin \frac{π}{2^{k}} \end{matrix} ⇔ \{ \begin{matrix}\frac{α_{i+1}}{r} & = & \frac{α_{i}}{r} cos \frac{π}{2^{i}} & + & \frac{β_{i}}{r} sin \frac{π}{2^{i}} \\ \frac{β_{i+1}}{r} & ≥ & \frac{β_{i}}{r} cos \frac{π}{2^{i}} & − & \frac{α_{i}}{r} sin \frac{π}{2^{i}} \\ −\frac{β_{i+1}}{r} & ≤ & \frac{β_{i}}{r} cos \frac{π}{2^{i}} & − & \frac{α_{i}}{r} sin \frac{π}{2^{i}} \\ 1 & = & \frac{α_{k}}{r} cos \frac{π}{2^{k}} & + & \frac{β_{k}}{r} sin \frac{π}{2^{k}} \end{matrix}

which means that L_{k} is nothing more than the homogenized polyhedral cone corresponding to D_{k}.

Let us now suppose (r, x_{1}, x_{2}) ∈ 𝕃^{2}. Equivalence (9.4) implies (\frac{x_{1}}{r} , \frac{x_{2}}{r}) ∈ B_{2}(1), which in turn implies by Corollary 9.1 that there exists a vector (α, β) ∈ ℝ^{2k} such that (\frac{x_{1}}{r} , α, \frac{x_{2}}{r} , β) belongs to D_{k}. Using the link (9.5), this last inclusion is equivalent to (r, x_{1}, rα, x_{2}, rβ) ∈ L_{k}, which means that (r, α_{0}, β_{0}) belongs to the projection of L_{k} on the subspace (r, α_{0}, β_{0}). We have thus shown that this projection is a relaxation of 𝕃^{2}, the first condition for it to be an ϵ-approximation of 𝕃^{2}.

Supposing now (r, x_{1}, x_{2}) belongs to the projection of L_{k}, there exists a vector (α, β) ∈ ℝ^{2k} such that (r, x_{1}, α, x_{2}, β) ∈ L_{k}. The equivalence (9.5) implies then that (\frac{x_{1}}{r} , \frac{α}{r}, \frac{x_{2}}{r} , \frac{β}{r}) ∈ D_{k}, which means that (\frac{x_{1}}{r} , \frac{x_{2}}{r}) belongs to the projection of D_{k} on its subspace (α_{0}, β_{0}). Using now Corollary 9.1, which states that this projection is an ϵ-approximation of B_{2}(1) with ϵ = cos(\frac{π}{2^{k}})^{−1} − 1, we can write that ∥(\frac{x_{1}}{r} , \frac{x_{2}}{r})∥ ≤ 1 + ϵ, which can be rewritten as ∥(x_{1}, x_{2})∥ ≤ (1 + ϵ)r, which is the exactly the second condition for this projection to be an ϵ-approximation of 𝕃^{2}.

The last task we have to accomplish is to check what happens in the case where r ≤ 0. Suppose (r, x_{1}, α, x_{2}, β) ∈ L_{k}. Looking at the definition of L_{k}, and using the same reasoning as in the proof of Theorem 9.2, it is straightforward to show that the variables α_{0} and β_{0} can only be equal to 0 when r = 0, and that they cannot satisfy the constraints if r < 0 (i.e. in the first case the set {P_{0}} is equal to {(0, 0)} while in the second case {P_{0}} = ∅). Since this is also the situation of the second-order cone 𝕃^{2}, our approximation is exact when r ≤ 0, and we can conclude that the projection of L_{k} on the subspace of its three variables (r, α_{0}, β_{0}) is an ϵ-approximation of the three dimensional second-order cone 𝕃^{2} with ϵ = cos(\frac{π}{2^{k}})^{−1} −1.

9.2.5 Reducing the approximation

Our polyhedral approximation L_{k} features 2k + 3 variables, 2k linear inequalities and k + 1 linear equalities. It is possible to reduce these numbers by pivoting out a certain number of variables. Namely, using the set of constraints α_{i+1} = α_{i} cos \frac{π}{2^{i}} + β_{i} sin \frac{π}{2^{i}} for 0 ≤ i < k, we can replace α_{k} by a linear combination of α_{k−1} and β_{k−1}, then replace α_{k−1} by a linear combination of α_{k−2} and β_{k−2}, and so on until all variables α_{i} have been replaced except α_{0} (which cannot and should not be pivoted out since it belongs to the projected approximation). The last equality r = α_{k} cos \frac{π}{2^{k}} + β_{k} sin \frac{π}{2^{k}} can also be used to pivot out β_{k}.

The resulting polyhedron has then k +2 variables (r, α_{0}, β_{0}, …, β_{k−1}), 2k linear inequalities and no linear equality. However, it should be noted that the constraint matrix describing the reduced polyhedron is denser than in the original approximation, i.e. it contains many more nonzero elements, as depicted on Figure 9.3 in the case k = 15 (which also mentions the number of nonzero elements in each case).

This denser constraint matrix has of course a negative impact on the efficiency of the algorithm used to solve the approximation problems, so that computational experiments are needed to decide whether this is enough to counterbalance the advantage of a reduced number of equalities and variables Indeed, preliminary testing on a few problems representative of the ones we are going to consider in Section 9.3 led us to the conclusion that pivoting out the variables is beneficial, leading roughly to a 20% reduction of computing times.

Figure 9.3: Constraint matrices for L_{15} and its reduced variant.

Another interesting remark can be made when we have to approximate a second-order cone whose components are restricted to be nonnegative. Namely, if we know beforehand that x_{1} and x_{2} cannot be negative, the polyhedral approximation of 𝕃^{2} can be reduced. Indeed, looking back at the proof of Theorem 9.2, we see that the set of points P_{2} is bounded by 2^{k−2} consecutive sides of the regular 2^{k}-side polyhedron (see for example the set {P_{2}} depicted in Figure 9.2). Combining this with the restriction that α_{2} and β_{2} are nonnegative, we have that the set {P_{2}} is exactly equal to the restriction of P_{2^{k}} to the positive orthant, and is thus a valid ϵ-approximation of 𝕃^{2} on this positive orthant with ϵ = cos(\frac{π}{2^{k}})^{−1} − 1. This observation leads to the formulation of a reduced polyhedral approximation L′_{k} defined by

\{(r, α_{2}, …, α_{k}, β_{2}, …, β_{k}) ∈ ℝ^{2k−1} | \{ \begin{matrix}α_{i+1} & = & α_{i} cos \frac{π}{2^{i}} & + & β_{i} sin \frac{π}{2^{i}} \\ β_{i+1} & ≥ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ −β_{i+1} & ≤ & β_{i} cos \frac{π}{2^{i}} & − & α_{i} sin \frac{π}{2^{i}} \\ r & = & α_{k} cos \frac{π}{2^{k}} & + & β_{k} sin \frac{π}{2^{k}} \end{matrix} ∀ 2 ≤ i < k\} ,

whose projection of the subspace of (r, α_{2}, β_{2}) approximates the nonnegative part 𝕃^{2}. This approximation features 2k − 1 variables, 2k − 4 linear inequalities and k − 1 linear equalities and can be reduced to k variables, 2k − 4 linear inequalities and no linear equality if we perform the pivoting described above.

At this stage, we would like to compare our approximation with the one presented in [BTN98]. Both of these feature the same accuracy ϵ = cos(\frac{π}{2^{k}})^{−1} − 1 (with parameter ν in [BTN98] equal to k − 1 in our setting). However, Ben-Tal and Nemirovski do not make explicit that the projection of their polyhedral approximation is equal to the regular 2^{k}-sided polyhedron in ℝ^{2} and only prove the corresponding accuracy result.

Table 9.1 compares the sizes of the polyhedral approximations in three cases: the original approximation, the reduced approximation where variables α_{i} are pivoted out and the nonnegative approximation L′_{k} (also with variables α_{i} pivoted out).

Table 9.1: Comparison of our approximation L_{k} with [BTN98].

Original  Reduced  Nonnegative
[BTN98] L_{k} [BTN98] L_{k} [BTN98] L′_{k}
Variables  2k + 3 2k + 3 k + 4 k + 2 k + 2  k
Inequalities 2k + 4  2k  2k + 4  2k  2k  2k − 4
Equalities  k − 1  k + 1  0  0  0  0

Our version uses 4 less inequality constraints in all three cases. It also features 2 more equality constraints in the original approximation, which turns out to be an advantage since it allows us to pivot out more variables in the reduced versions. Both the reduced and the nonnegative versions of L_{k} use 2 less variables than their counterparts in the original article of Ben-Tal and Nemirovski.

9.2.6 An approximation of 𝕃^{n}

We are now going to use the decomposition presented in Section 9.2.2 and our polyhedral approximation L_{k} for 𝕃^{2} to build an approximation for 𝕃^{n}. Recall that expression (9.3) decomposed 𝕃^{n} into ⌊n/2⌋ three-dimensional second-order cones 𝕃^{2} and a single larger cone 𝕃^{⌈ \frac{n}{2} ⌉}. Applying this decomposition recursively, we can decompose 𝕃^{⌈ \frac{n}{2} ⌉} into ⌊⌈n/2⌉/2⌋ secondorder cones 𝕃^{2} with a remaining larger cone 𝕃^{⌈⌈n/2⌉/2⌉}, which can be again decomposed into ⌊⌈⌈n/2⌉/2⌉/2⌋ cones 𝕃^{2}, etc.

Calling q_{k} the number of three-dimensional second-order cones appearing in the decomposition at each stage of this procedure and r_{k} the corresponding size of the remaining cone, we have initially q_{0} = 0, r_{0} = n and

q_{k} = ⌊\frac{r_{k−1}}{2} ⌋ and r_{k} = ⌈\frac{r_{k−1}}{2} ⌉ ∀ k > 0 .

Obviously, r_{k} is strictly decreasing and we must eventually end up with r_{k} equal to 2. Indeed, it is easy to see that 2^{i−1} < r_{k−1} ≤ 2^{i} implies 2^{i−2} < r_{k} ≤ 2^{i−1} and a simple recursive argument shows then that if 2^{m−1} < n ≤ 2^{m} we have 2^{m−k−1} < r_{k} ≤ 2^{m−k} and thus that r_{m−1} = 2. At this stage, the remaining second-order cone is 𝕃^{2}, which we can add to the decomposition in the last stage with q_{m} = 1 to have r_{m} = 0. Our decomposition has thus in total m stages. We also showed in Section 9.2.2 that the total of 𝕃^{2} cones in the final decomposition is equal to ∑_{i=1}^{m} q_{i} = n − 1, and we also note for later use that 2^{m−k} < r_{k−1} ≤ 2^{m−k+1} implies 2^{m−k−1} ≤ q_{k} ≤ 2^{m−k}.

We ask ourselves now what happens if each of the second-order cones appearing in this decomposition is replaced by an ϵ-approximation. Namely, suppose each of the ⌊n/2⌋ secondorder cones 𝕃^{2} in expression (9.3) is replaced by an ϵ^{(i)}-approximation (0 ≤ i ≤ ⌊n/2⌋), while the remaining larger cone is replaced by an ϵ′-approximation. We end up with the set

\{(r, x, y) ∈ ℝ_{+}×ℝ^{n+⌊ \frac{n}{2} ⌋} | (y_{i}, x_{2i−1}, x_{2i}) ∈ 𝕃_{ϵ^{(i)}}^{2}, 1 ≤ i ≤ ⌊\frac{n}{2}⌋, \begin{cases}(r, y) ∈ 𝕃_{ϵ′}^{⌈n/2⌉} & (n even) \\ (r, y, x_{n}) ∈ 𝕃_{ϵ′}^{⌈n/2⌉} & (n odd) \end{cases}\} ,

whose constraints are equivalent to

x_{2i−1}^{2} + x_{2i}^{2} ≤ (1 + ϵ^{(i)})^{2}y_{i}^{2}, 1 ≤ i ≤ ⌊\frac{n}{2}⌋, \begin{cases}∑i=1^{\frac{n}{2}} y_{i}^{2} & ≤ (1 + ϵ′)^{2}r^{2} (n even) \\ ∑i=1^{\frac{n}{2}} y_{i}^{2} + x_{n}^{2} & ≤ (1 + ϵ′)^{2}r^{2} (n odd) \end{cases} \} .

Ideally, we would like this decomposition to be an ϵ-approximation of 𝕃^{n}. We already know that it is a relaxation of 𝕃^{n}, since each approximation of 𝕃^{2} is itself a relaxation. We have thus to concentrate on the second condition defining an ϵ-approximation, ∥x∥ ≤ (1 + ϵ)r. Writing

∑_{i=1}^{2⌊n/2⌋} x_{i}^{2} ≤ ∑_{i=1}^{⌊n/2⌋}(1 + ϵ^{(i)})^{2}y_{i}^{2} ,

we would like to bound the quantity on the right hand-side. Unfortunately, we only know a bound on the sum of y_{i}^{2}’s, which forces us to write

∑_{i=1}^{2⌊n/2⌋} x_{i}^{2} ≤ (1 + max_{i} ϵ^{(i)})^{2} ∑_{i=1}^{⌊n/2⌋} y_{i}^{2} ⇒ ∑_{i=1}^{n} x_{i}^{2} ≤ (1 + max_{i} ϵ^{(i)})^{2}(1 + ϵ′)^{2}r^{2} .

This shows that our decomposition is an approximation of 𝕃^{n} with accuracy ϵ = (1 + max_{i} ϵ_{i})(1 + ϵ′). This immediately implies that there is no point in approximating with different accuracies the ⌊n/2⌋ small second-order cones 𝕃^{2} appearing in the decomposition, since only the largest of these accuracies has an influence on the resulting approximation for 𝕃^{n}. Applying now our decomposition recursively to the remaining cone, and choosing at each stage k a unique accuracy ϵ_{k} for all the 𝕃^{2} cones, we find that 1 + ϵ = ∏_{k=1}^{m}(1 + ϵ_{k}) ,, i.e. that the final accuracy of our polyhedral approximation is the product of the accuracies chosen at each stage of the decomposition (note that, unlike the situation for a single stage, there is no reason here to choose all ϵ_{k} accuracies to be equal to each other).

9.2.7 Optimizing the approximation

The previous section has shown how to build a polyhedral approximation of 𝕃^{n} and how its quality depends on the accuracy of the approximations used at each stage of the decomposition. Our goal is here to optimize these quantities, i.e. given a target accuracy ϵ for 𝕃^{n}, find the values of ϵ_{k} (1 ≤ k ≤ m) that lead to the smallest polyhedral approximation, i.e. the one with the smallest number of variables and constraints.

Let us suppose we use at stage k the approximation L_{u_{k}} with u_{k} + 2 variables and 2u_{k} linear inequalities (i.e. with variables α pivoted out of the formulation), which has an accuracy ϵ_{k} = cos(\frac{π}{2^{u_{k}}})^{−1} − 1. Recalling notation q_{k} for the number of cones 𝕃^{2} introduced at stage k of the decomposition, the final polyhedral approximation has thus an accuracy equal to ∏_{k=1}^{m} cos(\frac{π}{2^{u_{k}}})^{−1} with 2 ∑_{k=1}^{m} q_{k}u_{k} inequalities and n + ∑_{k=1}^{m} q_{k}u_{k} variables. Indeed, we have n original x_{i} variables and u_{k} additional variables for each of the q_{k} approximations at stage k, since the first two variables in these approximations are coming from the previous stage. We observe that the main quantity to be minimized is ∑_{k=1}^{m} q_{k}u_{k} for both the number of variables and inequalities, which leads to the following optimization problem:

σ_{n,ϵ} = min_{u∈ℕ^{m}} ∑_{k=1}^{m} q_{k}u_{k} s.t. ∏_{k=1}^{m} cos(\frac{π}{2^{u_{k}}})^{−1} ≤ 1 + ϵ .  (9.6)

A possible choice for variables u_{k} is to take them all equal. Plugging this unique value into the accuracy constraint, we readily find that u_{k} has to be equal to

u_{k} = ⌈log_{2}(π/ arccos(1 + ϵ)^{−1/m})⌉

and that when the dimension of the cone 𝕃^{n} (and thus m) tends to +∞, we have u_{k} = O (log \frac{m}{ϵ}) and σ_{n,ϵ} = O (n log \frac{m}{ϵ}).

This obviously does not lead to an optimal solution of (9.6). Indeed, since the number of approximations is decreasing as we move from one stage of the decomposition to the next, it is intuitively clear that trading a lower accuracy for first stages against a higher accuracy for the last stages will be beneficial, since the lowering of the number of variables and inequalities in the first stages will affect many more constraints than the increase of size for the last stages. This implies that the components u_{k} of any optimal solution of (9.6) will have to be in increasing order.

Finding a closed form optimal solution of (9.6) does not appear to be possible, but we can find a good suboptimal solution using some approximations. We first introduce variables v_{k} such that v_{k} = 4^{−u_{k}} ⇔ u_{k} = − log_{4} v_{k} and rewrite problem (9.6) as

σ_{n,ϵ} = min_{v∈ℝ^{m}} − log_{4} ∏_{k=1}^{m} v_{k}^{q_{k}} s.t. ∑_{k=1}^{m} log(cos(π\sqrt{v_{k}})^{−1}) ≤ log(1 + ϵ) .

Since u_{k} ≥ 2, we have π\sqrt{v_{k}} ≤ \frac{π}{4} and we can use the easily proven^{2} inequality

log(cos(x)^{−1}) ≤ (\frac{3x}{4})^{2}

valid for all 0 ≤ x ≤ \frac{π}{4} to write

σ_{n,ϵ} = − max_{v∈ℝ^{m}} log_{4} ∏_{k=1}^{m} v_{k}^{q_{k}} s.t. ∑_{k=1}^{m} v_{k} ≤ \frac{16}{9}π^{−2} log(1 + ϵ) = K(ϵ) ,

which is thus a restriction of our original problem. It amounts to maximizing a product of variables whose sum is bounded, a problem whose optimality conditions are well-known. In our case, they can written as

\frac{v_{1}}{q_{1}} = \frac{v_{2}}{q_{2}} = … \frac{v_{m}}{q_{m}} = \frac{∑_{k=1}^{m} v_{k}}{∑_{k=1}^{m} q_{k}} = \frac{K(ϵ)}{n − 1} ⇒ v_{k} = \frac{q_{k}}{n − 1}K(ϵ) ⇔ u_{k} = log_{4} \frac{n − 1}{q_{k}} − log_{4} K(ϵ) .

However u_{k} must be integer, so that we have to degrade this solution further and round it towards a larger integer. Using that fact that n − 1 ≤ 2^{m} and q_{k} ≥ 2^{m−k−1}, we have

\frac{n − 1}{q_{k}} ≤ \frac{2^{m}}{2^{m−k−1}} = 2^{k+1} ⇒ log_{4} \frac{n − 1}{q_{k}} ≤ log_{4} 2^{k+1} = \frac{k + 1}{2} ,

so that we can take u_{k} = ⌈\frac{k+1}{2}⌉ − ⌊log_{4} K(ϵ)⌋ as our suboptimal integer solution for (9.6).

2This inequality can be easily checked by plotting the graphs of its two sides on the interval [0 \frac{π}{4} ].

Let us plug now these values into the objective function σ_{n,ϵ}: we find

∑_{k=1}^{m} q_{k}u_{k} = ∑_{k=1}^{m} q_{k}⌈\frac{k + 1}{2}⌉ − ∑_{k=1}^{m} q_{k}⌊log_{4} K(ϵ)⌋

≤ ∑_{k=1}^{m} q_{k}(\frac{k}{2} + 1) − (n − 1)⌊log_{4} K(ϵ)⌋ (using ∑_{k=1}^{m} q_{k} = n − 1)

≤ ∑_{k=1}^{m} 2^{m−k}(\frac{k}{2} + 1) − (n − 1)⌊log_{4} K(ϵ)⌋ (using q_{k} ≤ 2^{m−k})
≤ 2^{m+1} − \frac{m}{2} − 2 − (n − 1)⌊log_{4} K(ϵ)⌋
≤ 4(n − 1) − (n − 1)⌊log_{4} K(ϵ)⌋ (using 2^{m−1} ≤ n − 1)
≤ (n − 1)⌈4 − log_{4} K(ϵ)⌉ = (n − 1)⌈4 − log_{4} \frac{16}{9} + log_{4} π^{2} − log_{4} log(1 + ϵ)⌉
≤ (n − 1)⌈5.3 − log_{4} log(1 + ϵ)⌉

(where we have used at the fourth line the fact that ∑_{k=1}^{m} 2^{m−k}(\frac{k}{2} +1) = 2^{m+1} − \frac{m}{2} −2, which is easily proved recursively). We can wrap this result into the following theorem:

Theorem 9.4. For every ϵ < \frac{1}{2}, there exists a polyhedron with no more than

2 + (n − 1)⌈5.3 − log_{4} log(1 + ϵ)⌉ = O(n log \frac{1}{ϵ}) variables

and

2 + 2(n − 1)⌈4.3 − log_{4} log(1 + ϵ)⌉ = O(n log \frac{1}{ϵ}) inequalities

whose projection on a certain subspace of n + 1 variables is an ϵ-approximation of the secondorder cone 𝕃^{n} ⊆ ℝ^{n+1}.

Proof. This a consequence of the previous derivation, which showed that choosing u_{k} = ⌈\frac{k+1}{2}⌉ − ⌊log_{4} K(ϵ)⌋ lead to an ϵ-approximation of 𝕃^{n} with n + σ_{n,ϵ} variables and 2σ_{n,ϵ} linear inequalities, with σ_{n,ϵ} = (n − 1)⌈5.3 − log_{4} log(1 + ϵ)⌉. However, the size of this polyhedron can be further reduced using L′_{k}, the polyhedral approximation of the nonnegative part of 𝕃^{2}. Indeed, looking at the decomposition (9.3), we see that all the y variables used in the second stage of the decomposition are guaranteed to be nonnegative, since we have in our approximation (y_{i}, x_{2i−1}, x_{2i}) ∈ L_{u_{1}} which implies y_{i} ≥ 0. This means that we can use for the second stage and the following our reduced approximation L′_{u_{k}}, known to be valid when its first two variables are restricted to the nonnegative orthant 𝕃^{2}, which uses 2 less variables and 4 less inequalities per cone. Since there are \frac{n}{2} cones in the first stage of the decomposition and n − 1 cone in total, we can use \frac{n}{2} − 1 reduced approximations L′, which give us a total saving of n − 2 variables and 2n − 4 constraints^{3}. Combining this with the value of σ_{n,ϵ}, we find that our approximation has 2+(n−1)⌈5.3−log_{4} log(1+ϵ)⌉ variables and 2+⌈4.3−log_{4} log(1+ϵ)⌉ inequalities.

3This reasoning was made for an even n. In the case of an odd n, we have one cone in the decomposition for which only the first variable is known to be nonnegative. It is possible to show that there exists a polyhedral approximation adapted to this situation that uses 1 less variable and 2 less inequalities than the regular approximation, which allows us to write exactly the same results as for an even n.

We also have to prove the asymptotic behaviour of σ_{n,ϵ} when n tends to infinity. Indeed, we have log(1 + ϵ) ≥ \frac{ϵ}{2} when ϵ < \frac{1}{2}, which implies − log_{4} log(1 + ϵ) ≤ log_{4} \frac{1}{ϵ}. This leads to ⌈5.3 − log_{4} log(1 + ϵ)⌉ = O(log \frac{1}{ϵ}), which is enough to prove the theorem.

This result is better than the one we previously obtained choosing all u_{k}’s equal to each other: indeed, we had in that case σ_{n,ϵ} = O(n log \frac{m}{ϵ}) = O(n log \frac{log n}{ϵ}) while we have here O(n log \frac{1}{ϵ}). For a fixed accuracy ϵ, our first choice translates into σ_{n,ϵ} = O (n log log n) when n tends to infinity while our optimized choice of u_{k}’s leads to σ_{n,ϵ} = O (n), which is better. We note however that if we fix n and let ϵ tends to 0, the asymptotic behaviour is the same in both cases , namely we have σ_{n,ϵ} = O(log \frac{1}{ϵ}).

Ben-Tal and Nemirovski achieve essentially the same result in [BTN98], albeit in the special case when n is a power of two. Our proof has the additional advantage of providing a closed form for the parameters u_{k} as well as for the total size of the polyhedral approximation, for all values of n. They also prove that the number of inequalities of a ϵ-approximation of 𝕃^{n} must be greater that O(n log \frac{1}{ϵ}), i.e. that the order of the result of Theorem 9.4 is not improvable.

9.2.8 An approximation of second-order cones optimization

The previous sections have proven the existence of a polyhedral approximation of the secondorder cone with a moderate size (growing linearly with the dimension of the cone and the logarithm of the accuracy). However, we have to point out that these polyhedrons are not strictly speaking approximations of the second-order cone: more precisely, it is their projection on a certain subspace than is an ϵ-approximation of 𝕃^{n}.

This does not pose any problem when trying to approximate a second-order cone optimization problem with linear optimization. Let us suppose we want to approximate problem (9.1), which we recall here for convenience,

inf c^{T} x s.t. Ax = b and x^{k} ∈ 𝕃^{n_{k}} ∀k = 1, 2, …, r  (9.1)

with ϵ-approximations of the second order cones 𝕃^{n_{k}}. Theorem 9.4 implies the existence of a polyhedron

Q_{k} = \{(x^{k}, y^{k}) ∈ ℝ^{n_{k}+1} × ℝ^{O(n_{k} log \frac{1}{ϵ})} | A_{k}(x^{k}, y^{k})^{T} ≥ 0\}

with

A_{k} ∈ ℝ^{O(n_{k} log \frac{1}{ϵ})×O(n_{k} log \frac{1}{ϵ})} ,

whose projection on the subspace (r, x) is an ϵ-approximation of 𝕃^{n_{k}}, which allows us to write the following linear optimization problem^{4}

min c^{T} x s.t. Ax = b and A_{k}(x^{k}, y^{k})^{T} ≥ 0 ∀k = 1, 2, …, r .  (9.7)

We note that the fact that our approximations are projections is handled in a seamless way by this formulation: the only difference with the use of a direct approximation of the cones 𝕃^{n_{k}} is the addition of the auxiliary variables y^{k} to the formulation. This problem features ∑_{k=1}^{r} O(n_{k} log \frac{1}{ϵ}) = O(n log \frac{1}{ϵ}) variables, m equality constraints and ∑_{k=1}^{r} O(n_{k} log \frac{1}{ϵ}) = m + O(n log \frac{1}{ϵ}) homogeneous inequality constraints. We also point out as a minor drawback of this formulation the fact that it involves irrational coefficients, namely the quantities sin \frac{π}{2^{i}} and cos \frac{π}{2^{i}} occurring in the definition of L_{k}. However, it rational coefficients are really needed (for example if one wants to work with a complexity model based on exact arithmetic), it is possible to replace those quantities with rational approximations while keeping an essentially equivalent accuracy for the resulting polyhedral approximation, i.e. featuring the same asymptotic behaviour.

4We could replace the inf of problem (9.1) by a min since it is well-known that linear optimization problems always attain their optimal objectives, see Chapter 3.

To conclude this section, we are going to compare the algorithmic complexity of solving problem (9.1) either directly or using our polyhedral approximation. The best complexity obtained so far^{5} for solving a linear program with v variables up to accuracy ϵ is O(v^{3.5} log(\frac{1}{ϵ})) arithmetic operations (using for example a short-step path-following method, see Chapter 1). In our case, assuming we solve the approximate problem (9.7) up to the same accuracy that the one used to approximate the second-order cones, this leads to a complexity equal to O(n^{3.5} log(\frac{1}{ϵ})^{4.5}).

On the other hand, solving problem (9.1) can be done using O(\sqrt{r} n^{3} log(\frac{1}{ϵ})) arithmetic operations, using for example a potential reduction approach, see e.g. [LVBL98]. If r = O (1), i.e. if the number of cones used in the formulation is bounded, the second complexity is better, both if n → +∞ or ϵ → 0. However, if r = O (n), which means that the dimension of the cones used in the formulation is bounded, both complexity become equivalent from the point of view of the dimension n, but the second one is still better when letting the accuracy tend to 0. We conclude that the direct solving of (9.1) as a second-order cone problem is superior from the point of view of algorithmic complexity. The purpose of the second part of this chapter will be to test whether this claim is also valid for computational experiments.

9.2.9 Accuracy of the approximation

The linearizing scheme for second-order cone optimization presented in the previous section is based on a polyhedral approximation whose accuracy is guaranteed in the sense of Definition 9.2. It is important to realize that this bound on the accuracy of the approximation does not imply a bound on the accuracy of the solutions (or the objective value) of the approximated problem.

Indeed, let us consider the following set:

\{(r, x_{1}, x_{2}) ∈ ℝ^{3} | r − x_{2} = \frac{1}{2} and (r, x_{1}, x_{2}) ∈ 𝕃^{2}\} .

This set can be seen as the feasible region of a second-order cone problem. Using the fact that

x_{1}^{2} + x_{2}^{2} ≤ r^{2} ⇔ x_{1}^{2} + x_{2}^{2} ≤ (x_{2} + \frac{1}{2})^{2} ⇔ x_{1}^{2} − \frac{1}{4} ≤ x_{2} ,

we find that the projection of this set on the subspace (x_{1}, x_{2}) is the epigraph of the parabola x ↦ x^{2} − \frac{1}{4}. Let us now replace 𝕃^{2} by the polyhedral approximation L_{k}. Since the resulting set will be polyhedral, its projection on the subspace (x_{1}, x_{2}) will also be polyhedral, and we can deduce without difficulties that it is the epigraph of a piecewise linear function, as shown by Figure 9.4 (depicting the cases k = 1, 2, 3 and 4).

5Using standard linear algebra and without partial updating.

Figure 9.4: Linear approximation of a parabola using L_{k} for k = 1, 2, 3, 4.

Because a polyhedron has a finite number of vertices, this piecewise linear function must have a finite number of segments. Considering the rightmost piece, i.e. whose x_{1} values span an interval of the type [α + ∞[, it is obvious that it cannot approximate the parabola with a guaranteed accuracy. Indeed, the difference between the approximation and the parabola grows quadratically on this segment, which means that even the ratio of variable x_{2} between the parabola and its linear approximation is not bounded.

Let us now consider the following parameterized family of second-order cone optimization problems

min x_{2} s.t. x_{1} = λ, r − x_{2} = \frac{1}{2} and (r, x_{1}, x_{2}) ∈ 𝕃^{2} ,  (PB_{λ})

which is using the same feasible set as above with an additional constraint fixing the variable x_{1} to λ. Denoting the optimal objective value of (PB_{λ}) by p^{∗}(λ), we have in light of the previous discussion that p^{∗}(λ) = λ^{2} − \frac{1}{4}. However, we also showed that the optimal objective value p_{k}^{∗}(λ) of the approximated problem

min x_{2} s.t. x_{1} = λ, r − x_{2} = \frac{1}{2} and (r, x_{1}, x_{2}, y) ∈ L_{k}

must be a piecewise linear function of λ with a finite number of segments. Indeed, simple computations^{6} show that the endpoints of these segments occur for

λ = \frac{sin(iθ)}{2 cos \frac{θ}{2} − 2 cos(iθ)} for i = 1, 2, …, 2^{k} − 1 with θ = \frac{π}{2^{k−1}} ,

which shows that p^{∗}(λ) is linear as soon as λ ≥ _{\frac{sin θ}{2 cos \frac{θ}{2} −2 cos θ}}.

The discrepancy between the real optimum p^{∗}(λ) and the approximated optimum p_{k}^{∗}(λ) is thus unbounded when λ goes to infinity. Moreover, we have that the relative accuracy of p_{k}^{∗}(λ) tends to 1, the worst possible value, i.e.

\frac{p^{∗}(λ) − p_{k}^{∗}(λ)}{p^{∗}(λ)} → 1 .

Another interesting feature of this small example is that performing a complete parametric analysis for parameter λ ranging from −∞ to +∞ would lead to 2^{k} −1 different break points.

We conclude that we cannot give an a priori bound on the accuracy of the optimal objective value of the linear approximation of a second-order cone optimization problem (this remark is also valid for accuracy of the optimal solution itself, since we have in our example (PB_{λ}) that the optimal value of x_{2} is equal to p^{∗}).

9.3 Computational experiments

In this section, we present computational experiments with an implementation of the linearizing scheme for second-order cone optimization we have just described.

9.3.1 Implementation

The computer used to conduct those experiments is an Intel 500 MHz Pentium III with 128 megabytes of memory. We chose to use the MATLAB programming environment, developed by The MathWorks, for the following reasons:

⋄ MATLAB is a flexible and modular environment for technical computing, two very important characteristics when developing research code. Although MATLAB may be somehow slower than a pure C or FORTRAN approach, we think that this loss of performance is more than compensated by the ease of development (especially from the point of view of graphic capabilities and debugging). Moreover, the critical (i.e. time consuming) parts of the algorithms can be coded separately in C or FORTRAN and used in MATLAB via MEX files (this is the approach taken by the solvers we mention below), which allows a well designed MATLAB program to be nearly as efficient as an equivalent pure C or FORTRAN program.

6Simply observe that the extremal rays of L_{k} obey to the relation x_{2} = x_{1} tan iθ with i = 1, 2, …, 2^{k} and θ = π/2^{k−1}.

⋄ Efficient interior-point solvers are available on the MATLAB platform. Indeed, we used in our experiments

– The MOSEK optmization toolbox for MATLAB by EKA Consulting ApS, a fullfeatured optimization package including a simplex solver and primal-dual interiorpoint solvers for linear optimization, convex linearly and quadratically constrained optimization, second-order cone optimization, linear least square problems, linear l_{1} and l_{∞}-norm optimization and geometric and entropy optimization [AA99, ART00]. When compared with the standard optimization toolbox from MATLAB, MOSEK is particularly efficient on large-scale and sparse problems. MOSEK can be downloaded for research and evaluation purposes at http://www.mosek.com.

– SeDuMi by Jos Sturm [Stu99b], another primal-dual interior-point solver which is able to handle linear, second-order cone and semidefinite optimization problems. SeDuMi is designed to take into account sparsity and complex values, and has the advantage of dealing with the very important class of semidefinite optimization, but is a little more restrictive than MOSEK concerning the input format, since problems must be entered in the standard conic form (9.1). SeDuMi can be

downloaded at http://www.unimaas.nl/~sturm/.

The main routines we implemented are the following (source code is available in the appendix):

⋄ PolySOC2(k) generates the polyhedron L_{k} with accuracy ϵ_{k} = cos(\frac{π}{2^{u_{k}}})^{−1} −1. Variables α_{i} are pivoted out, so that this routine returns a polyhedron with k + 2 variables and 2k inequalities. An optional parameter is available to use the reduced approximation L′_{k}, valid on the nonnegative restriction of 𝕃^{2}.

⋄ Steps(q, e) computes the optimal choice for the size of the cones at each stage of the decomposition of 𝕃^{n}. Indeed, q contains our vector q (i.e. the number of cones at each stage) and e is the target accuracy ϵ.

⋄ PolySOCN(n, e) generates a e-approximation of 𝕃^{n}. It uses the output of PolySOC2 and the optimal sizes for the cones computed by Steps.

⋄ PolySOCLP(p, e) linearizes the second-order cone optimization problem p, replacing each second-order cone constraint with a polyhedral e-approximation using PolySOCN and outputting a linear optimization problem.

The procedure Steps we implemented features some improvements when compared with the theory we presented in the previous section. Indeed, Theorem 9.4 shows that the choice u_{k} = ⌈\frac{k+1}{2}⌉ − ⌊log_{4} K(ϵ)⌋ leads to a polyhedron of size O(n log \frac{1}{ϵ}), but is not optimal for two reasons:

⋄ We approximated the formula giving the accuracy of the approximation to derive u_{k} (namely, we used log(cos(x)^{−1}) ≤ (\frac{3x}{4})^{2}).

⋄ The optimal solution for this approximated accuracy was not guaranteed to be integer and had to be rounded to the smallest greater integer.

However, one can easily improve this choice in practice as follows. Let us suppose theory predicts some optimal values v_{k} for the sizes of the cones at stage k, which have to be rounded to ⌈v_{k}⌉. Because of this rounding, the actual accuracy of the approximation will be much better than our target ϵ. Recalling now that this accuracy is equal to (1 + ϵ_{1})(1 + ϵ′), where ϵ_{1} is the accuracy of the cones in the first stage, and is thus equal to cos(\frac{π}{2^{⌈v_{1}⌉}})^{−1} − 1, and ϵ′ is the accuracy from the cone modelled by all the remaining stages 𝕃^{⌈n/2⌉}, we can compute an upper bound for ϵ′, according to

(1 + ϵ_{1})(1 + ϵ′) ≤ ϵ ⇔ ϵ′ ≤ \frac{ϵ}{1 + ϵ_{1}} − 1

which will be better (i.e. higher) than in the theoretical derivation since it takes into account the exact accuracy ϵ_{1} of the first stage, rounding included. We can now apply this procedure to the second stage, i.e. computing a theoretical value for ϵ_{2} and an upper bound on the accuracy of 𝕃^{⌈⌈n/2⌉/2⌉}, and so on, obtaining in the end a smaller polyhedral approximation, since the required accuracies for the cones at every stage (except the first one) are higher and hence need less constraints and variables.

Still, this improved rounding does not address the first reason why our u_{k}’s are not optimal, the fact that we do not optimize the actual formula for the accuracy. Since it seems impossible to deal with it in closed form, we implemented a dynamic programming approach to optimize it. This algorithm uses the theoretical suboptimal solution described above (including the improved rounding procedure) to provide bounds on the optimal solution and therefore reduce the computing time.

Figure 9.5 presents the graphs of the size (measured by σ_{n,ϵ}) of our approximations in two situations: fixed dimension (n = 50, 200, 800) with accuracy ranging from 10^{−1} to 10^{−8} and fixed accuracy (ϵ = 10^{−2}, 10^{−5}, 10^{−8}) with dimension ranging from 10 to 1000. The asymptotic behaviour of σ_{n,ϵ} is very clear on these graphs: we have a linear increase when n tends to +∞ for a fixed accuracy and a logarithmic increase when ϵ tends to 0 for a fixed dimension (since the first graph has a logarithmic scale of abscissas).

Finally, in order to give an idea of the efficiency of our improved rounding procedure and dynamic programming resolution, we provide in Table 9.2 the value of σ_{n,ϵ} for different strategies, using a target accuracy ϵ = 10^{−8}.

Table 9.2: Different approaches to optimize the size of a 10^{−8}-approximation of 𝕃^{n}

n Theory All equal (rounded) Theory (rounded) Dynamic programming
10  171  139  141  139
100  1881  1584  1552  1537
1000 18981  15987  15688  15522
10000 189981  160140  157063  155392

The first column represents the theoretical value σ_{n,ϵ} = (n − 1)⌈5.3 − log_{4} log(1 + ϵ)⌉, the second column describes the choice of all u_{k}’s, equal to each other, albeit using the improved rounding procedure presented above, the third column reports the choice of the theoretical Figure 9.5: Size of the optimal approximation versus accuracy (left) and dimension (right).

value for u_{k}, this time with the improved rounding procedure, and the last column gives the true optimal value via our dynamic programming approach. We observe that our iterative rounding procedure improves the theoretical value of σ_{n,ϵ} in a noticeable way, lowering it by approximately 15%. The differences between the last three columns are less important, the dynamic programming approach giving a few additional percents of decrease in the size of the approximation.

9.3.2 Truss-topology design

We first tested our linearizing scheme for second-order cone optimization on a series of truss topology design problems. A truss is a structure composed of elastic bars connecting a set of nodes, like a railroad bridge or the Eiffel tower. The task consists in determining the size (i.e. the cross sectional areas) of the bars that lead to the stiffest truss when submitted to a set of forces, subject to a total weight limit. The problem we want to solve here is a multi-load truss topology design problem, which means we are simultaneously considering a set of k loading scenarios. This problem can be formulated as follows (see [BTN94]):

min ∑_{i=1}^{n} σ_{i} s.t. ∥(q_{i1}, …, q_{ik})∥ ≤ σ_{i} ∀1 ≤ i ≤ n and B (q_{1j} ; q_{2j} ; ⋮ ; q_{nj}) = f_{j} ∀1 ≤ j ≤ k , (TTD)

where n is the number of bars, k is the number of loading, vector σ ∈ ℝ^{n} and matrix Q ∈ ℝ^{n×k} are the design variables, B ∈ ℝ^{m×n} is a matrix describing the physical configuration of the truss and f_{j} ∈ ℝ^{m}, 1 ≤ j ≤ k are vectors of forces describing the loadings scenarios.

It is easily cast as a second-order cone problem in the form (9.1), since the norm constraints can be modelled as (σ_{i}, q_{i1}, …, q_{ik}) ∈ 𝕃^{k} for all 1 ≤ i ≤ n. Indeed, we have that the variables x ∈ ℝ^{k(n+1)}, the objective c ∈ ℝ^{k(n+1)} and the equality constraints Ax = b with A ∈ ℝ^{km×k(n+1)} and b ∈ ℝ^{km} are given by

x = ( σ ; q_{11} ; q_{21} ; ⋮ ; q_{nk}) , c = (1^{n×1} ; 0 ; 0 ; ⋮ ; 0) , A = (\begin{matrix}0^{m×n} & B \\ 0^{m×n} & & B \\ ⋮ & & & ... \\ 0^{m×n} & & & & B \end{matrix}) and b = (f_{1} ; f_{2} ; ⋮ ; f_{k})

to give the following second-order cone optimization problem equivalent to (TTD):

min (1^{n×1} ; 0 ; 0 ; ⋮ ; 0)^{T} ( σ ; q_{11} ; q_{21} ; ⋮ ; q_{nk}) s.t. \{ \begin{matrix}(\begin{matrix}0^{m×n} & B \\ 0^{m×n} & & B \\ ⋮ & & & ... \\ 0^{m×n} & & & & B \end{matrix}) ( σ ; q_{11} ; q_{21} ; ⋮ ; q_{nk}) = (f_{1} ; f_{2} ; ⋮ ; f_{k}) \\ (σ_{i}, q_{i1}, …, q_{ik}) ∈ 𝕃^{k} for all 1 ≤ i ≤ n \end{matrix} . (TP)

This allows us to write a dual problem in the form (9.2) in a straightforward manner:

max ∑_{j=1}^{k} f_{j}^{T} y_{j} s.t. \{ \begin{matrix}(\begin{matrix}0^{m×n} & 0^{m×n} & ⋯ & 0^{m×n} \\ B^{T} \\ & B^{T} \\ . & . & . \\ & & & B^{T} \end{matrix}) (y_{σ} ; y_{1} ; ⋮ ; y_{k}) + ( σ^{∗} ; q_{11}^{∗} ; q_{21}^{∗} ; ⋮ ; q_{nk}^{∗}) = (1^{n×1} ; 0 ; 0 ; ⋮ ; 0) \\ (σ_{i}^{∗}, q_{i1}^{∗}, …, q_{ik}^{∗}) ∈ 𝕃^{k} for all 1 ≤ i ≤ n \end{matrix} (TD)

where σ^{∗} ∈ ℝ^{n}, Q^{∗} ∈ ℝ^{n×k}, y_{σ} ∈ ℝ^{m} and y_{j} ∈ ℝ^{m} for all 1 ≤ j ≤ k are the dual variables. Variables σ^{∗} and Q^{∗} can be pivoted out of the formulation using the linear constraints

σ^{∗} = 1^{n×1} and q_{ij}^{∗} = −b_{i}^{T} y_{j} ∀1 ≤ i ≤ n, 1 ≤ j ≤ k

(where b_{i} ∈ ℝ^{m} is the i^{th} column of B), which gives then

max ∑_{j=1}^{k} f_{j}^{T} y_{j} s.t. (1, −b_{i}^{T} y_{1}, …, −b_{i}^{T} y_{k}) ∈ 𝕃^{k} for all 1 ≤ i ≤ n

and finally

max ∑_{j=1}^{k} f_{j}^{T} y_{j} s.t. ∑_{j=1}^{k}(b_{i}^{T} y_{j})^{2} ≤ 1 for all 1 ≤ i ≤ n ,  (TQC)

which is a convex quadratically constrained problem with a linear objective. We can thus solve a truss-topology design problem in at least three different manners: either solving the secondorder cone optimization problems (TP) or (TD) or solving the quadratically constrained problem (TQC).

The problems we used for our computational experiments were randomly created using a generator developed by A. Nemirovski. Given three integers p, q and k, it produced the

Table 9.3: Dimensions of the truss-topology design problems.

Problem description  Formulation  Primal  Dual
2 × 2 grid with 2 loads 5 cones 𝕃^{2}  15 × 8  23 × 15
2 × 2 grid with 4 loads 5 cones 𝕃^{4}  25 × 16  41 × 25
2 × 2 grid with 8 loads 5 cones 𝕃^{8}  45 × 32  77 × 45
2 × 2 grid with 16 loads 5 cones 𝕃^{16}  85 × 64  149 × 85
2 × 2 grid with 32 loads 5 cones 𝕃^{32}  165 × 128  293 × 165
4 × 4 grid with 2 loads 114 cones 𝕃^{2}  342 × 48  390 × 342
4 × 4 grid with 4 loads 114 cones 𝕃^{4}  570 × 96  666 × 570
4 × 4 grid with 6 loads 114 cones 𝕃^{6}  798 × 144  942 × 798
6 × 6 grid with 2 loads 615 cones 𝕃^{2} 1845 × 120 1965 × 1845
6 × 6 grid with 4 loads 615 cones 𝕃^{4} 3075 × 240 3315 × 3075
8 × 8 grid with 2 loads 1988 cones 𝕃^{2} 5964 × 224 6188 × 5964

matrix B and vectors f_{j} corresponding to k loading scenarios for a truss using a 2-dimensional p × q nodal grid, with n ≈ \frac{1}{2}p^{2}q^{2} and m ≈ 2pq. We tested 11 combinations of parameters p, q and k. The dimensions of the corresponding problems are reported in Table 9.3 (the last two columns report the number of variables × the number of constraints). We see that the last problems involve a fairly large number of small second-order cones.

Polyhedral approximations of these problems were computed for three different accuracies, namely ϵ = 10^{−2}, 10^{−5} and 10^{−8}. The dimensions of the resulting linear optimization problems are reported in Table 9.4. It is interesting to note that problems with accuracy 10^{−8} are only approximately three times larger than problems with accuracy 10^{−2} and 50% larger than problems with accuracy 10^{−5} (with several dozens of thousands of variables for the largest among them).

Before we present computing times, we have to mention a special feature of this class of problems. Contrary to the general assertion that is stated in Section 9.2.9, it is possible to give here an estimation of the quality of the optimum objective value of the approximated problem. Indeed, let us call t^{∗} the optimal objective value of problem (TTD) and t_{ϵ}^{∗} the optimal objective value of the approximated problem with accuracy ϵ. Since our approximation is a relaxation, we obviously have t_{ϵ}^{∗} ≤ t^{∗}, and the optimal solution of the approximated problem (Q_{ϵ}^{∗}, σ_{ϵ}^{∗}) is not necessarily feasible for the original problem. However, Definition 9.2 of a ϵ-approximation of a second-order cone implies in our case that

∥(q_{ϵ,i1}^{∗}, …, q_{ϵ,ik}^{∗})∥ ≤ (1 + ϵ)σ_{ϵ,i}^{∗} ∀1 ≤ i ≤ n ,

which means that (Q_{ϵ}^{∗}, (1 + ϵ)σ_{ϵ}^{∗}) is feasible for the original problem, with an objective value equal to (1 + ϵ)t_{ϵ}^{∗}. Since we must have then t^{∗} ≤ (1 + ϵ)t_{ϵ}^{∗}, we conclude that

\frac{t^{∗}}{1 + ϵ} ≤ t_{ϵ}^{∗} ≤ t^{∗} ⇔ 0 ≤ \frac{t^{∗} − t_{ϵ}^{∗}}{t^{∗}} ≤ \frac{ϵ}{1 + ϵ} ,

i.e. we have a bound on the relative accuracy of our approximated optimum objective value.

Table 9.4: Dimensions of the approximated problems (primal above, dual below).

\begin{matrix}p × q × k & ϵ = 10^{−2} & ϵ = 10^{−5} & ϵ = 10^{−8} \\ 2 × 2 × 2 & 35 × 58 & 60 × 108 & 85 × 158 \\ 2 × 2 × 4 & 85 × 146 & 160 × 296 & 235 × 446 \\ 2 × 2 × 8 & 200 × 352 & 370 × 692 & 545 × 1042 \\ 2 × 2 × 16 & 420 × 744 & 795 × 1494 & 1165 × 2234 \\ 2 × 2 × 32 & 860 × 1528 & 1635 × 3078 & 2410 × 4628 \\ 4 × 4 × 2 & 798 × 1188 & 1368 × 2328 & 1938 × 3468 \\ 4 × 4 × 4 & 1938 × 3060 & 3648 × 6480 & 5358 × 9900 \\ 4 × 4 × 6 & 3306 × 5388 & 6156 × 11088 & 9006 × 16788 \\ 6 × 6 × 2 & 4305 × 6270 & 7380 × 12420 & 10455 × 18570 \\ 6 × 6 × 4 & 10455 × 16230 & 19680 × 34680 & 28905 × 53130 \\ 8 × 8 × 2 & 13916 × 20104 & 23856 × 39984 & 33796 × 59864 \end{matrix}

\begin{matrix}p × q × k & ϵ = 10^{−2} & ϵ = 10^{−5} & ϵ = 10^{−8} \\ 2 × 2 × 2 & 43 × 65 & 68 × 115 & 93 × 165 \\ 2 × 2 × 4 & 101 × 155 & 176 × 305 & 251 × 455 \\ 2 × 2 × 8 & 232 × 365 & 402 × 705 & 577 × 1055 \\ 2 × 2 × 16 & 484 × 765 & 859 × 1515 & 1229 × 2255 \\ 2 × 2 × 32 & 988 × 1565 & 1763 × 3115 & 2538 × 4665 \\ 4 × 4 × 2 & 846 × 1482 & 1416 × 2622 & 1986 × 3762 \\ 4 × 4 × 4 & 2034 × 3534 & 3744 × 6954 & 5454 × 10374 \\ 4 × 4 × 6 & 3450 × 6042 & 6300 × 11742 & 9150 × 17442 \\ 6 × 6 × 2 & 4425 × 7995 & 7500 × 14145 & 10575 × 20295 \\ 6 × 6 × 4 & 10695 × 19065 & 19920 × 37515 & 29145 × 55965 \\ 8 × 8 × 2 & 14140 × 25844 & 24080 × 45724 & 34020 × 65604 \end{matrix}

Table 9.5: Computing times to solve truss-topology problems using different approaches.

QCO  SOCO  10^{−2}  10^{−5}  10^{−8}
p × q × k  (D) (P) (D)  (D’) (P) (D) (P) (D)  (P) (D)
2 × 2 × 2  0.00 0.00 0.00  0.58 0.01 0.00 0.01 0.02  0.04 0.03
2 × 2 × 4  0.01 0.01 0.00  0.10 0.02 0.02 0.05 0.05  0.11 0.12
2 × 2 × 8  0.01 0.01 0.01  0.12 0.05 0.05 0.14 0.14  0.33 0.34
2 × 2 × 16 0.01 0.02 0.01  0.19 0.10 0.11 0.37 0.37  0.85 0.83
2 × 2 × 32 0.02 0.03 0.03  0.35 0.26 0.27 0.87 0.90  1.96 1.86
4 × 4 × 2  0.09 0.06 0.29  0.60 0.30 0.21 0.73 0.69  1.54 1.52
4 × 4 × 4  0.11 0.13 0.74  1.74 1.52 0.92 3.19 2.86  9.30 5.61
4 × 4 × 6  0.20 0.26 2.22  5.03 3.86 2.09 7.98 5.50 13.50 10.84
6 × 6 × 2  0.47 0.61 1.96 30.13 2.54 1.88 21.59 5.16 34.95 10.45
6 × 6 × 4  1.28 1.32 6.48 475.73 19.08 7.82 40.80 23.89 396.04 43.43
8 × 8 × 2  4.30 2.76 11.81 339.66 12.08 8.89 53.29 24.55 127.68 48.42

We generated three random problems for each of the 11 combinations of parameters (p, q, k) presented in Table 9.3, and report in Table 9.5 the average computing time in seconds. Each column in this table corresponds to a different way to solve the truss-topology design problem:

a. the first column reports computing times using MOSEK on the quadratically constrained formulation (TQC),

b. the following three columns report computing times using MOSEK on the primal and the dual second-order cone formulations (TP)–(TD) in columns (P) and (D), as well as the results of SeDuMi on the dual formulation in column (D’).

c. the last six columns report computing times using the interior-point code in MOSEK to solve the polyhedral approximations of the primal and dual second-order cone problems (TP)–(TD) with three different accuracies.

Our first constatation is that solving the quadratically constrained formulation (TQC) and the primal second-order cone formulation (TP) directly are the two fastest methods (with similar computing times). The quadratically constrained formulation has less variables and constraints, but this advantage seems to be counterbalanced by a more efficient second-order cone solver.

Solving the dual second-order cone formulation (TP) directly is also very fast with a 2 × 2 nodal grid but noticeably slower on the larger problems (3 to 8 times slower). This is most probably due to the greater dimensions of the problem. The SeDuMi solver is much less efficient to solve these dual problems, and is really slow on the three largest problems.

Let us now look at the approximated problems. First of all, we checked whether the accuracy of the optimum approximated objective was below the theoretical bound \frac{ϵ}{1+ϵ}, since rounding errors in the computations and handling of irrational coefficients could affect this result. Unsurprisingly, the accuracy was below the theoretical threshold for all experiments. Computing times are worse than for the direct approaches, even with the low accuracy 10^{−2}. The difference grows up to one or two orders of magnitude for the larger problems.

We also observe that despite slightly greater dimensions, solving the approximated dual problem is more efficient than solving the primal problem, especially with the largest problems. The reasons for this behaviour, which is opposite to the situation for direct resolutions, are unclear to us, but could be related to sparsity issues.

Finally, let us mention that we also tried to solve the linear approximations using the simplex algorithm instead of an interior-point method. This lead to surprisingly bad computing times: for example, solving problem 4 × 4 × 4 with accuracy 10^{−2} using the MOSEK^{7} simplex code took 21.57 seconds, instead of 1.52 seconds with the interior-point algorithm. We believe this disastrous behaviour of the simplex algorithm is due to the presence of an exponential number of vertices in the approximation, which leads to very slow progress.

9.3.3 Quadratic optimization

Second-order cone formulations of truss-topology design problems feature a relatively large number of small cones. Since our approximation procedure has not proven to be more efficient than direct methods on these problems, we would like to turn our attention to the opposite configuration, i.e. a small number of large cones. We are going to show that convex quadratic optimization can be formulated such as to meet this requirement.

More specifically, we are going to consider linearly constrained convex quadric optimization problems. Such problems can be formulated as

min \frac{1}{2}x^{T} Qx + c^{T} x + c_{0} s.t. l_{c} ≤ Ax ≤ u_{c} and l_{x} ≤ x ≤ u_{x} ,  (QO)

where x ∈ ℝ^{n} denotes the vector of design variable. The objective is defined by a matrix Q ∈ ℝ^{n×n}, required to be positive semidefinite to ensure convexity of the problem, a vector c ∈ ℝ^{n} and a scalar c_{0} ∈ ℝ. Variables are bounded by two vectors l_{x} ∈ ℝ^{n} and u_{x} ∈ ℝ^{n} (note that some components of l_{x} or u_{x} can be equal to −∞ or +∞ if a variable has no lower or upper bound). Finally, the linear constraints are described by a matrix A ∈ ℝ^{m×n} and two vectors l_{c} ∈ ℝ^{m} and u_{c} ∈ ℝ^{m} (with the same remark holding about possible infinite values for some components of l_{c} and u_{c}).

In order to model problem (QO) with a second-order cone formulation, we first write the Cholevsky factorization of matrix Q. Indeed, we have Q = L^{T} L with L ∈ ℝ^{k×n}, where k ≤ n is the rank of Q. Introducing a vector of auxiliary variables z ∈ ℝ^{k} such that z = Lx, we have that x^{T} Qx = x^{T} L^{T} Lx = (Lx)^{T} Lx = z^{T} z, which allows us to write the following problem:

min \frac{r + v}{2} +c^{T} x+c_{0} s.t. (r, v, z) ∈ 𝕃^{k+1}, r−v = 1, l_{c} ≤ Ax ≤ u_{c} and l_{x} ≤ x ≤ u_{x} . (QO’)

7In order to make sure that this behaviour was not caused by a flaw in the MOSEK simplex solver, we performed a similar comparison with the CPLEX solver, which lead to the same conclusion.

It is indeed equivalent to (QO), since the conic constraint (r, v, z) ∈ 𝕃^{k+1} combined with the equality r − v = 1 leads to

v^{2} + ∑_{i=1}^{k} z_{i}^{2} ≤ r^{2} ⇔ ∑_{i=1}^{k} z_{i}^{2} ≤ r^{2} − v^{2} ⇔ z^{T} z ≤ (r − v)(r + v) ⇔ z^{T} z ≤ r + v ,

which is why the quadratic term \frac{1}{2}x^{T} Qx in the objective of (QO) could be replaced by \frac{r+v}{2} in (QO’).

The problems we tested come from the convex quadratic optimization library QPDATA, collected by Maros and Mészáros [MM99]. As for our tests with truss-topology design problems, we decided to formulate approximations with three different accuracies 10^{−2}, 10^{−5} and 10^{−8}. Table 9.6 lists for each problem its original size (variables × constraints), the number of nonzero elements in the constraint matrix A, the number of nonzero elements in the upper triangular part of Q and the size (variables × constraints) of each of the three polyhedral approximations.

Table 9.7 reports computing times (in seconds) needed to solve these convex quadratic optimization problems in three different ways:

a. the first column reports computing times measured using MOSEK directly on the original quadratic formulation (QO),

b. the following two columns report computing times needed to solve the second-order cone formulation (QO’) of these problems with MOSEK and SeDuMi (in columns labelled (SOCO) and (SOCO’) respectively),

c. the last three columns report computing times using MOSEK to solve the polyhedral approximations of the second-order cone problem (QO’) with three different accuracies.

Once again, the direct approach, i.e. solving (QO), is the most efficient method. Solving these problems with a second-order cone formulation is slower, especially on larger problems. Using SeDuMi instead of MOSEK degrades further the computing times.

It is also manifest that the linear approximations take more time than the direct approach to provide a solution, even with the lowest accuracy 10^{−2} (however, we note that this low accuracy approximation is faster than the SeDuMi resolution on a few problems).

Although we only tested small-scale and medium-scale problems, it is pretty clear from the trend present for the last problems that large-scale problems would also be most efficiently solved directly as convex quadratic optimization problems.

We pointed out in Section 9.2.9 that our bound on the accuracy of the polyhedral approximation did not imply anything on the quality of the optimum of the approximated problems. Indeed, Table 9.8 reports the relative accuracy for a few representative approximated problems. Some problems (GENHS28, DUALC5) behave very well, with a relative accuracy well below the target accuracy. Other problems (GOULDQP2, MOSARQP1) have higher relative accuracies, but still decreasing when the target accuracy is decreased. Problem CVXQP3S shows a worse

Table 9.6: Statistics for the convex quadratic optimization problems.

Name  Size ANZ QNZ  Size 10^{−2}  Size 10^{−5}  Size 10^{−8}
TAME  2 × 1  2  3  9 × 13  14 × 23  19 × 33
HS21  2 × 1  2  2  14 × 22  24 × 42  34 × 62
ZECEVIC2  2 × 2  4  1  9 × 14  14 × 24  19 × 34
HS35  3 × 1  3  5  20 × 31  35 × 61  50 × 91
HS35MOD  3 × 1  3  5  20 × 31  35 × 61  50 × 91
HS52  5 × 3  7  7  29 × 46  49 × 86  69 × 126
HS76  4 × 3  10  6  28 × 46  48 × 86  68 × 126
HS51  5 × 3  7  7  29 × 46  49 × 86  69 × 126
HS53  5 × 3  7  7  29 × 46  49 × 86  69 × 126
S268  5 × 5  25  15  34 × 57  59 × 107  84 × 157
HS268  5 × 5  25  15  34 × 57  59 × 107  84 × 157
GENHS28  10 × 8  24  19  61 × 100  106 × 190  151 × 280
LOTSCHD  12 × 7  54  6  47 × 70  76 × 128  106 × 188
HS118  15 × 17  39  15  99 × 169  174 × 319  248 × 467
QPCBLEND  83 × 74 491  83  545 × 914  959 × 1742  1373 × 2570
CVXQP2S  100 × 25  74 386 647 × 1020  1135 × 1996  1624 × 2974
CVXQP1S  100 × 50 148 386 647 × 1045  1135 × 2021  1624 × 2999
CVXQP3S  100 × 75 222 386 647 × 1070  1135 × 2046  1624 × 3024
QPCBOEI2 143 × 166 1196 143 939 × 1614  1652 × 3040  2366 × 4468
DUALC5  8 × 278 2224  36  54 × 61  94 × 441  134 × 521
PRIMALC1  230 × 9 2070 229 1505 × 2329  2646 × 4611  3789 × 6897
PRIMALC5  287 × 8 2296 286 1878 × 2903  3304 × 5755  4732 × 8611
DUAL4  75 × 1  75 2799  493 × 761  867 × 1509  1241 × 2257
GOULDQP2 699 × 349 1047 697 2635 × 3872  4370 × 7342 6107 × 10816
DUAL1 85 × 1 85 3558 558 × 861 982 × 1709 1406 × 2557
PRIMALC8  520 × 8 4160 519 3410 × 5268 5996 × 10440 8586 × 15620
GOULDQP3 699 × 349 1047 1395 4584 × 7420 8062 × 14376 11546 × 21344
DUAL2 96 × 1 96 4508 632 × 976 1110 × 1932 1589 × 2890
MOSARQP2 900 × 600 2390 945 5911 × 9721 10395 × 18689 14887 × 27673

Table 9.7: Computing times to solve convex quadratic optimization problems

Name  QO SOCO SOCO’ 10^{−2} 10^{−5} 10^{−8}
TAME  0.06  0.11  0.46 0.01 0.01 0.02
HS21  0.00  0.01  0.17 0.01 0.01 0.02
ZECEVIC2 0.01  0.00 0.101 0.00 0.00 0.03
HS35 0.00 0.00 0.1 0.01 0.01 0.02
HS35MOD 0.00  0.03 0.361 0.01 0.01 0.02
HS52  0.00  0.01  0.09 0.01 0.02 0.03
HS76  0.00  0.00  0.11 0.00 0.02 0.04
HS51  0.00  0.01  0.08 0.00 0.02 0.04
HS53  0.01  0.00  0.09 0.00 0.02 0.03
S268  0.02  0.01  0.20 0.01 0.04 0.07
HS268  0.01  0.00  0.20 0.01 0.04 0.07
GENHS28 0.00  0.01  0.08 0.01 0.03 0.07
LOTSCHD 0.02  0.01  0.16 0.02 0.03 0.08
HS118  0.00  0.01  0.25 0.05 0.11 0.17
QPCBLEND 0.03  0.12  0.83 0.41 1.43 2.75
CVXQP2S 0.02  0.23  1.34 0.58 1.91 3.58
CVXQP1S 0.03  0.14  1.55 0.57 1.88 4.47
CVXQP3S 0.05  0.21  1.86 0.60 2.32 3.93
QPCBOEI2 0.08  0.59  4.01 1.26 5.53 11.55
DUALC5  0.01  0.01  1.56 0.04 0.04 0.09
PRIMALC1 0.04  0.91  2.17 1.27 7.93 17.85
PRIMALC5 0.03  0.83  0.69 3.15 9.49 20.83
DUAL4  0.04  0.13  0.57 0.39 0.83 1.42
GOULDQP2 1.21  0.41  1.87 3.54 10.14 20.65
DUAL1  0.06  0.18  0.72 0.72 1.66 2.53
PRIMALC8 0.08  4.72  4.32 3.97 30.32 50.99
GOULDQP3 0.98 12.16  6.32 9.16 38.44 74.15
DUAL2 0.07 0.16 0.93 0.88 2.03 3.03
MOSARQP2 0.38  1.64 8.512 7.69 39.03 76.59

Table 9.8: Relative accuracy of the optimum of some approximated problems.

Name  10^{−2}  10^{−5}  10^{−8}
GENHS28  5.2e-4 8.8e-7 7.6e-10
HS21  3.3e-10 3.3e-10 3.3e-10
CVXQP3S  6.7e-3 7.2e-4 4.7e-4
DUALC5  3.9e-3 7.2e-7 3e-10
GOULDQP2 5.2e-1 5.6e-3 2.1e-5
MOSARQP2 8.6e-1 2.5e-4 8.5e-7

behaviour, with virtually no improvement between the second and the third approximation. Finally, HS21 is a toy problem with the surprising property that its approximation is exact at any accuracy.

We were able to compute these relative accuracies because the true optimal objective values were known by other means. In a real-world situation where such a piece information would not be available, it would be still possible to estimate roughly this accuracy. Indeed, since our approximation is a relaxation, we have that the approximated optimal objective p_{ϵ}^{∗} is lower than the true optimum p^{∗}. On the other hand, the optimal solution x_{ϵ}^{∗} of the approximation must be feasible, since it satisfies the linear constraints. Computing the objective function corresponding to this solution, i.e. letting p′_{ϵ}^{∗} = \frac{1}{2}x_{ϵ}^{∗}^{T} Qx_{ϵ}^{∗} + c^{T} x_{ϵ}^{∗} + c_{0}, we have finally that p_{ϵ}^{∗} ≤ p^{∗} ≤ p′_{ϵ}^{∗}, which allows us to estimate a posteriori the true optimum objective value.

However, in the special case where the objective is purely quadratic, i.e. when c = 0 and c_{0} = 0, it is possible to slightly modify the formulation so that we have a bound on the accuracy of the objective^{8}. Indeed, letting again z = Lx, we add this time the conic constraint (r, z) ∈ 𝕃^{k}, which implies to z^{T} z ≤ r^{2} ⇔ x^{T} Qx ≤ r^{2}. We can now choose r as our objective, which is equivalent to minimizing \sqrt{x^{T} Qx} and is obviously the same thing as minimizing the true quadratic objective \frac{1}{2}x^{T} Qx. This leads to a situation that is very similar to the case of truss-topology design problems, and one can show without difficulties that this approximated problem provides an estimation of the true optimum with a relative accuracy equal to (\frac{ϵ}{1+ϵ})^{2}.

9.4 Concluding remarks

In this chapter, we presented a polyhedral approximation of the second-order cone originally developed by Ben-Tal and Nemirovski [BTN98]. Our presentation features several improvements, including smaller dimensions for the approximation, a more transparent proof of its correctness, complete developments valid for any size of the second-order cone (i.e. not limited to powers of two) and explicit constants in the derivation of a theoretical bound on the size of the approximation (Theorem 9.4).

8 A similar improvement can be made in the case when c belongs to the column space of L^{T}, the Cholevsky factor of Q. However, we have been unable to generalize this construction to the case where c does not belong to this column space, e.g. for an objective equal to x_{1}^{2} + x_{2}.

This scheme was implemented in MATLAB and optimized as much as possible. Indeed, we developed several approaches to reduce the size of the resulting linear problems (including pivoting out some variables and using dynamic programming to choose the best accuracies for each stage of the decomposition). Our experiments mainly showed that solving the original second-order cone problems or alternative equivalent formulations is more efficient than solving the linear approximations, even at low accuracies. On a side note, we noticed that these approximate problems are particularly difficult to solve with the simplex algorithm.

However, we would like to point out this approximating scheme can still prove very useful in certain well-defined circumstances, such as a situation where a user is equipped with a solver that is only able to solve linear optimization problems. In this case, this procedure provides him with an inexpensive and relatively straightforward way to test improved versions of his linear models that make use of second-order cones.

Moreover, we have to admit that we tested two very specific classes of second-order cone optimization problems for which either a simplified formulation or a well-understood dedicated algorithm was available. It might well be possible that this linearizing scheme becomes competitive for other types of difficult (i.e. that cannot be simplified and for which no dedicated solver is available) second-order cones optimization problems.

We would also like to insist on the fact that it is not possible to guarantee a priori the accuracy of a linear approximation of a general second-order optimization problem (see the example in Section 9.2.9). It is nevertheless possible to provide such a bound in some special cases (e.g. truss-topology design problems or convex quadratic optimization problems with a pure quadratic objective).

It is worth to point out that a straightforward modification of our polyhedral approximation of 𝕃^{2} can lead to a restriction instead of a relaxation of second-order cone optimization problems. This would then provide an upper bound instead of a lower bound on the true optimum objective value, and optimal solutions of the approximate problems would always be feasible for the original problem. However, this approach can be problematic in some cases since it might happen that the approximated problem is infeasible, even if the original problem admits some feasible solutions.

An interesting topic for further research is the generalization of the polyhedral approximation of 𝕃^{2} or, more precisely, of the unit ball B_{2}(1), to other convex sets. Indeed, finding a similar polyhedral approximation for a set like {(x_{1}, x_{2}) ∈ ℝ^{2} | |x_{1}|^{p} + |x_{2}|^{p} ≤ 1} with p > 1, i.e. the unit ball for the p-norm, would lead to linearizing scheme for other classes of convex problems, such as l_{p}-norm optimization (see Chapter 4). However, it is unclear to us at this stage whether this goal is achievable or not, since the symmetry of the standard unit ball, which is not present for other norms, seems to play a great role in the construction of the approximation.

Part IV Conclusions

187

Concluding remarks and future research directions

We give here some concluding remarks about the research presented in this thesis, highlighting our personal contributions and hinting at some possible directions for further research (we however refer the reader to the last section of each chapter for more detailed comments).

Interior-point methods

Chapters 1 and 2 presented a survey of interior-point methods for linear optimization and a self-contained overview of the theory of self-concordant functions for structured convex optimization. We contributed some new results in Chapter 2, namely the computation of the optimal complexity of the short-step method and the improvement of a very useful Lemma to prove self-concordancy. We also gave a detailed explanation of why the definition of selfconcordancy that is most commonly used nowadays is the best possible.

A very promising research direction in this area consists in investigating other types of barriers functions that lead to polynomial-time algorithms for convex optimization, possibly using the single condition (2.18) instead of the two inequalities (2.2) and (2.3) that characterize a self-concordant function.

Conic duality

Chapter 3 presented the framework of conic optimization and the associated duality theory, which is heavily used in the rest of this thesis. The approach we take in Chapters 4–6 to study l_{p}-norm and geometric optimization and give simplified proofs of their duality properties is completely new. The corresponding convex cones L^{p}, G^{n}, G_{2}^{n} were to the best of our knowledge never studied before.

Chapter 7 generalizes our conic formulations of geometric and l_{p}-norm optimization with the notion of separable cone and is the culminating point of our study of convex problems with a nonsymmetric dual. We believe that most of the structured convex optimization that one can encounter in practice can be formulated within this framework (with the notable exceptions of second-order cone and semidefinite optimization).

It is obvious that much more research has to be done in this area. First of all, it would be highly desirable to study the duality properties relating the primal-dual pair of separable problems (SP)–(SD). Proving weak duality and strong duality in the presence of a Slater point should be straightforward. Moreover, we believe the zero duality gap property can probably also be proved (possibly with some minor technical assumptions), because of the inherent separability that is present in the definition of the K^{f} cone (i.e. the fact that all the functions that are used within this definition are scalar functions).

Another promising approach consists in generalizing the self-concordant barrier we designed for the L^{p} cone to the whole class of separable cones K^{f} and implementing the corresponding interior-point algorithms. Based on the results of existing conic solvers for linear, second-order and semidefinite optimization, our feeling is that the conic approach could lead to significant improvements in computational efficiency over more traditional methods.

Approximations

Chapter 8 demonstrated that it is possible to approximate geometric optimization using l_{p}- norm optimization. Despite the large amount of similarities between these two problems that was noticed by several authors, it is to the best of our knowledge the first time that such a strong link between these two classes of problems is presented.

Finally, Chapter 9 described a linearizing scheme for second-order cone optimization first introduced in [BTN98]. Our presentation features several improvements over the original construction, such as smaller dimensions for the polyhedral approximation, a more transparent proof of its correctness, complete developments valid for any size of the second-order cone (i.e. not limited to powers of two) and explicit constants in the derivation of a theoretical bound on the size of the approximation. We also contributed a careful implementation of this procedure using the MATLAB programming environment.

Although the computational experiments we conducted tend to show that solving the approximated problems is not as efficient as solving directly the original problem, we would like to stress the nonintuitive fact, demonstrated in this chapter, that it is possible, albeit with a relative loss of efficiency, to solve second-order cone and quadratic optimization problems with a linear optimization solver. Another interesting topic for further research in this area would be to generalize the principle of this polyhedral approximation to other types of convex sets.

Part V Appendices

191

APPENDIX A

An application to classification

We present here a summary of our research on the application of semidefinite optimization to classification, which was the topic of our master’s thesis [Gli98b ].

A.1 Introduction

Machine learning is a scientific discipline whose purpose is to design computer procedures that are able to perform classification tasks. For example, given a certain number of medical characteristics about a patient (e.g. age, weight, blood pressure), we would like to infer automatically whether he or she is healthy or not.

A special case of machine learning problem is the separation problem, which asks to find a way to classify patterns that are known to belong to different well-defined classes. This is equivalent to finding a procedure that is able to recognize to which class each pattern belongs. The obvious utility of such a procedure is its use on unknown patterns, in order to determine to which one of the classes they are most likely to belong.

In this chapter, we present a new approach for this question based on two fundamental ideas: use ellipsoids to perform the pattern separation and solve the resulting problems with semidefinite optimization.

A.2 Pattern separation

Let us suppose we are faced with a set of objects. Each of these objects is completely described by an n-dimensional vector. We call this vector a pattern. To each component in this vector corresponds in fact a numerical characteristic about the objects. We assume that the only knowledge we have about an object is its pattern vector.

Let us imagine there is a natural way to group those objects into c classes. The pattern separation problem is simply the problem of separating these classes, i.e. finding a partition of the whole pattern space ℝ^{n} into c disjoint components such that the patterns associated to each class belong to the corresponding component of the partition.

The main use for such a partition is of course classification: suppose we have some wellknown objects that we are able to group into classes and some other objects for which we don’t know the correct class. Our classification process will take place as follows:

a. Separate the patterns of well-known objects. This is called the learning phase^{1}.

b. Use the partition found above to classify the unknown objects. This is called the generalization phase.

We might ask ourselves what is a good separation. A good algorithm should of course be able to separate correctly the well-known objects, but is only really useful if it classifies correctly the unknown patterns. The generalization capability is thus the ultimate criteria to judge a separation algorithm.

We list here a few examples of common classification tasks.

⋄ Medical diagnosis. This is one of the most important applications. The pattern vectors represent various measures of a patient’s condition (e.g. age, temperature, blood pressure, etc. ). We want here to separate the class of ill people from the class of healthy people.

⋄ Species identification. The pattern vector represent various characteristics (e.g. colour, dimensions) of a plant or animal. Our objective is to classify them into different species.

⋄ Credit screening. A company is trying to evaluate applicants for a credit card. The pattern contains information about the customer (e.g. type of job, monthly income, owns a house) and the goal is to identify for which applicants it is financially safe to give a credit card.

Ellipsoid representation. The main idea of this chapter is to use ellipsoids to separate our classes. Assuming we want to separate two classes of patterns^{2}, this means that we would like to compute a separating ellipsoid such that the points from one class belong to the interior of the ellipsoid while the points from the other class lie outside of this ellipsoid. Let us explain this idea with Figure A.1.

1Some authors refer to it as supervised learning phase. In fact, one may want to separate patterns without knowing a priori the classes they belong to, which is then called unsupervised learning. This is in fact a clustering problem, completely different from ours, and won’t be discussed further in this work.

2It is shown in [Gli98b] that we can restrict our attention to the problem of separating two classes without loss of generality.

Figure A.1: A bidimensional separation problem.

This example is an easy bidimensional separation problem taken from a species classification data set (known as Fisher’s Iris test set), using only the first two characteristics. The patterns from the first class appear as small circles, while the other class appear as small crosses. Computing a separating ellipsoid leads to the situation depicted on Figure A.2.

Figure A.2: A separating ellipsoid.

We decided to use ellipsoids for the following reasons:

⋄ We expect patterns from the same class to be close to each other. This suggests enclosing them in some kind of hull, possibly a ball. But we also want our procedure to be scaling invariant. This is why we use the affine deformations of balls, which are the ellipsoids.

⋄ Ellipsoids are the simplest convex sets (besides affine sets, which obviously do not fit our purpose).

⋄ The set of points lying between two parallel hyperplanes is a (degenerate) ellipsoid. This means our separation procedures will generalize procedures that use a hyperplane to separate patterns.

⋄ We know that some geometrical problems involving ellipsoids can be modelled using semidefinite optimization (this is due to the fact that an ellipsoid can be conveniently described using a positive semidefinite matrix).

Separating patterns. Our short presentation has avoided two difficulties that may arise with a pattern separation algorithm using ellipsoids, namely

a. Most of the time, the separating ellipsoid is not unique. How do we choose one ?

b. It may happen that there exists no separating ellipsoid.

Both of these issues can be addressed with the use optimization. Each ellipsoid is a priori a feasible solution. The objective function of our program will measure how well this ellipsoid separates our points. Ideally, non separating ellipsoids should have a high objective value (since we minimize our objective), while separating ellipsoids should have a lower objective value. With this kind of formulation, the conic program will always give us a solution, even when there is no separating ellipsoid.

We have thus to find an objective function that adequately represents the quality of the ellipsoid separation.

A.3 Maximizing the separation ratio

Let us consider the simple example depicted on Figure A.3: we want to include the small circles in an ellipsoid in order to obtain the best separation from the small crosses. A way to express this is to ask for two different separating ellipsoids. We want these ellipsoids to share the same center and axis directions (i.e. we want them to be geometrically similar), but the second one will be larger by a factor ρ, which we will subsequently call the separation ratio. Figure A.4 shows such a pair of ellipsoids with a ρ equal to \frac{3}{2}.

We now use the separation ratio to assess the quality of the separation: the higher the value of ρ, the better the separation. Our goal will be to maximize ρ over the set of separating ellipsoids. Figure A.5 shows the optimal pair of ellipsoids, with the maximal ρ equal to 1.863. However, we don’t need two ellipsoids, so we finally partition the pattern

A.3 – Maximizing the separation ratio  197

Figure A.3: A simple separation problem. Figure A.4: A pair of ellipsoids with ρ equal to \frac{3}{2}.

198  A. An application to classification

Figure A.5: The optimal pair of separating ellipsoids. Figure A.6: The final separating ellipsoid. space using an intermediate ellipsoid whose size is the mean size of our two ellipsoids, as depicted on Figure A.6.

It is possible to use semidefinite optimization to model the problem of finding the pair of ellipsoids with the best separation ratio. However, a straightforward formulation does not work because it leads to non-convex constraints and a technique of homogenization has to be introduced. We refer the reader to [Gli98b] for a thorough description of this formulation featuring the relevant mathematical details.

A.4 Concluding remarks

We have sketched in this chapter the principles of pattern separation using ellipsoids. It is obviously possible to enhance the basic method we presented in several different ways (for example to handle the case where the patterns cannot be completely separated by an ellipsoid). Three variants of this method are indeed described in [Gli98b] (minimum volume method, maximum sum method and minimum squared sum method).

We also refer the reader to [Gli98b] for the presentation and analysis of extensive computational results involving these methods on standard test sets. The main conclusion that can be drawn from this study is that these methods provide a viable way to classify patterns. As far as comparison with other classification procedures is concerned, it is fair to say that separating patterns using ellipsoids with semidefinite optimization occasionally delivers excellent results (significantly better than any other existing procedure) and gives competitive error rates on the majority of data sets.

To conclude this chapter, we mention that this approach has been recently applied to the problem of predicting the success or failure of students at their final exams.

Indeed, using only the results of preliminary tests carried out by first-year undergraduate students in late November, our separating ellipsoid is able to predict with a 11% error rate which students are going to pass and be allowed to enter the second-year, a decision which in fact depends on a series of exams that occur 2, 5 and even in some cases 7 months later (see the forthcoming report [DG00] for a complete description of these experiments).

APPENDIX B

Source code

We provide here the source code of the main routines used in the linearizing scheme for second-order cone optimization described in Chapter 9.

function Epsilon = Accuracy(Steps);
% Accuracy   Compute the accuracy of a polyhedral SOC approximation
%            Epsilon = Accuracy(Steps) returns the accuracy of a polyhedral
%            approximation of a second-order cone using a pyramidal
%            construction based on approximations of 3-dimensional SOC,
%            where Steps contains the number of steps used for the
%            approximation made at each level of the pyramidal construction.
Epsilon = 1/prod(cos(pi * (1/2) .^Steps)) - 1;

function Levels = Levels(SizeCone)
% Levels    Computes the size of each level in the pyramidal approximation.
%           Levels = Levels(SizeCone) computes the number of cones
%           needed at each level in the pyramidal construction leading
%           to the polyhedral approximation of a second-order cone.

202  B. Source code

Levels = []; while SizeCone > 1
   Half = floor(SizeCone/2);
   Levels = [Levels Half];
   SizeCone = SizeCone - Half;
end

function theSteps = Steps(Levels, Epsilon, Method)
% Steps    Computes the number of steps for each level of the approximation
%          theSteps = Steps(Levels, Epsilon, Method) computes the number of
%          steps for each of the Levels in order to get accuracy equal to
%          Epsilon with the specified Method:
%            'AllEqual' -> number of steps is the same for each level
%            'Theory'   -> use formula with theoretical bound 'n log(1/e)'
%            'Optimal'  -> compute lowest possible total number of steps
if nargin < 3
   Method = 'Optimal';
end switch Method case 'AllEqual'
   theSteps = ceil(log2(pi/acos((1+Epsilon)^(-1/length(Levels)))));
   if length(Levels) > 1
      D = Accuracy(theSteps);
      theSteps = [Steps(Levels(1:end-1), (Epsilon-D)/(1+D), 'AllEqual') theSteps];
   end
case 'Theory'
   theSteps = ceil(log2(sum(Levels)/Levels(end)*9/16*pi^2/log(1+Epsilon))/2);
   if length(Levels) > 1
      D = Accuracy(theSteps);
      theSteps = [Steps(Levels(1:end-1), (Epsilon-D)/(1+D), 'Theory') theSteps];
   end
case 'Optimal'
   if length(Levels) == 1
      theSteps = Steps(1, Epsilon, 'AllEqual');
   else
      AE = Steps(Levels, Epsilon, 'AllEqual');
      TH = Steps(Levels, Epsilon, 'Theory');
      UpperBound = floor(min(AE*Levels', TH*Levels')/sum(Levels));
      LowerBound = Steps(1, Epsilon, 'AllEqual');
      theSteps = []; BestSize = inf;
      index = LowerBound;
      while index <= UpperBound
         D = Accuracy(index);
         S = [index Steps(Levels(2:end), (Epsilon-D)/(1+D), 'Optimal')];
         if S*Levels' < BestSize
            theSteps = S;
            BestSize = theSteps*Levels';
            UpperBound = min(UpperBound, floor(BestSize/sum(Levels)));
         end
         index = index + 1;

\underline{203}

     end
   end
otherwise
   error('Unknown method');
end

function resLP = PolySOC2(Steps, SkipSteps)
% PolySOC2  Computes a polyhedral approximation of the 3-dimensional Lorentz cone
%           resLP = PolySOC2(Steps, SkipSteps) computes a polyhedral approximation
%           of the 3-dimensional SOC using the Ben-Tal/Nemirovski construction with
%           a number of steps equal to Steps. A number of the first steps of the
%           construction can be skipped using the optional parameter SkipSteps.
%           The resulting approximation will have:
%                 n+2 variables (i.e. n-1 additional variables),
%                 2n  inequality constraints,
%           where n is the total number of steps in the construction (i.e.
%           Steps-SkipSteps). There are also two global options available:
%             - useRestriction to use a restriction of the SOC instead of a
%               relaxation,
%             - doNotPivotOut to stop pivoting out variables from the equality
%               constraints, which gives n more variables and n equality
%               constraints but a more sparse constraint matrix.
% Global options
global doNotPivotOut useRestriction;
persistent PolySOC2Cache;
if nargin < 2
   SkipSteps = 0;
else
   Steps = Steps-SkipSteps;
end
if [Steps+1 SkipSteps+1] <= size(PolySOC2Cache) & ...
      ~isempty(PolySOC2Cache{Steps+1, SkipSteps+1})
   resLP = PolySOC2Cache{Steps+1, SkipSteps+1};
   return;
end
Angles = pi * (1/2).^(SkipSteps+(0:Steps))';
indexX = repmat([1 1 1 2 2 2 3 3 3], Steps, 1) + repmat((0:3:3*(Steps-1))', 1, 9);
indexY = repmat([1 2 3 1 2 4 1 2 4], Steps, 1) + repmat((1:2:2*(Steps)-1)', 1, 9);
indexVal = [ cos(Angles(1:end-1))  sin(Angles(1:end-1)) -ones(Steps, 1) ...
             sin(Angles(1:end-1)) -cos(Angles(1:end-1)) -ones(Steps, 1) ...
            -sin(Angles(1:end-1))  cos(Angles(1:end-1)) -ones(Steps, 1) ];
if ~isempty(useRestriction) & useRestriction
   rootCoef = cos(Angles(end));
else
   rootCoef = 1;
end
A = sparse([indexX(:) ; 3*(Steps) + [1;1;1]], ...

        [indexY(:) ; 2*(Steps) + [2;3] ; 1], ...
        [indexVal(:) ; cos(Angles(end)) ; sin(Angles(end)) ; -rootCoef]);
if isempty(doNotPivotOut) | ~doNotPivotOut
for index = 1:Steps % alpha variables
   A = A + A(:, 3+index) * A(2*index-1, :);
   A(2*index-1, :) = [];
   A(:, 3+index) = [];
end
A = A - 1/A(end,end) * A(:, end) * A(end, :); % last beta_k variable
A(end, :) = [];
A(:, end) = [];
resLP = lp([], A, [-inf*ones(2*Steps, 1)], zeros(2*(Steps), 1));
else
resLP = lp([], A, [repmat([0 ; -inf ; -inf], Steps, 1) ; 0], ...
           zeros(3*(Steps)+1, 1));
end
PolySOC2Cache{Steps+1, SkipSteps+1} = resLP;

function [resLP, theAccuracy, theSteps] = PolySOCN(SizeCone, Epsilon)
% PolySOC2N  Computes a polyhedral approximation of a second-order cone
%            [resLP, theAccuracy, theSteps] = PolySOCN(SizeCone, Epsilon) computes
%            a polyhedral approximation with accuracy Epsilon of SOC of dimension
%            SizeCone (not counting the root) using a pyramidal construction
%            involving (SizeCone-1) 3-dimensional SOC approximations. theAccuracy
%            will contain the resulting accuracy (smaller or equal to Epsilon)
%            while theSteps provides the number of steps used for the approximation
%            at each level of the pyramidal construction.
switch SizeCone
case 0             % Special case: linear program, not handled by this construction
resLP = lp([], 1, 0);
theAccuracy = 0;
theSteps = [];
case 1             % Special case: linear program, not handled by this construction
resLP = lp([], [1 -1;1 1], [0 0]');
theAccuracy = 0;
theSteps = [];
otherwise
theLevels = Levels(SizeCone);
theSteps = Steps(theLevels, Epsilon, 'Optimal');
theAccuracy = Accuracy(theSteps);
CurrentVars = 1+(1:SizeCone);
resLP = lp([], zeros(0, SizeCone+1));
index = 1;
OddLeft = mod(SizeCone, 2);
for index = 1:length(theLevels)
   if index == 1
      addLP = PolySOC2(theSteps(index));

   [addLP, baseVars, rootVars] = DupPolySOCN(addLP, 2, theLevels(index));
elseif OddLeft & theLevels(index-1) ~= 2*theLevels(index)
   OddLeft = 0;
   addLP = PolySOC2(theSteps(index), 2);
   [addLP, baseVars, rootVars] = DupPolySOCN(addLP, 2, theLevels(index)-1);
   oddLP = PolySOC2(theSteps(index), 1);
   rootVars = [1 rootVars+dims(oddLP, 2)];
   baseVars = [2 3 baseVars+dims(oddLP, 2)];
   addLP = add(oddLP, addLP, []);
else
   addLP = PolySOC2(theSteps(index), 2);
   [addLP, baseVars, rootVars] = DupPolySOCN(addLP, 2, theLevels(index));
end
reOrder = NaN*ones(1, dims(addLP, 2));
reOrder(baseVars) = CurrentVars(end-theLevels(index)*2+1:end);
CurrentVars = [CurrentVars(1:end-theLevels(index)*2) dims(resLP, 2) + ...
   rootVars-(0:2:2*theLevels(index)-2)];
if index == length(theLevels)
   reOrder(1) = 1;
end
resLP = add(resLP, addLP, reOrder);
end
end

function [resLP, baseVars, rootVars] = DupPolySOCN(theLP, SizeCone, N);
% DupPolySOCN  Concatenate polyhedral approximations of second-order cones.
%              [resLP, baseVars, rootVars] = DupPolySOCN(theLP, SizeCone, N)
%              computes a concatenation of N polyhedral approximations of
%              a SizeCone-dimensional second-order cone contained in theLP.
%              rootVars contains the indices of the N root cone variables, while
%              baseVars contains the indices of the N*SizeCone other cone variables.
if N == 0
rootVars = [];
baseVars = [];
resLP = lp;
else
rootVars = 1;
baseVars = 2:SizeCone+1;
resLP = theLP;
nSteps = floor(log2(N));
N = N - 2^nSteps;
for index = nSteps-1:-1:0
Delta = dims(resLP, 2);
resLP = add(resLP, resLP, []);
baseVars = [baseVars Delta+baseVars];
rootVars = [rootVars Delta+rootVars];
if N >= 2^index

         N = N - 2^index;
         Delta = dims(resLP, 2);
         resLP = add(resLP, theLP, []);
         baseVars = [baseVars Delta+(2:SizeCone+1)];
         rootVars = [rootVars Delta+1];
      end
   end
end
% Alternate recursive version :
% if N == 0
%    rootVars = [];
%    baseVars = [];
%    resLP = lp;
% elseif N == 1
%    rootVars = 1;
%    baseVars = 2:SizeCone+1;
%    resLP = theLP;
% elseif mod(N, 2) == 0
%    [resLP baseVars rootVars] = DupPolySOCN(theLP, SizeCone, N/2);
%    Delta = dims(resLP, 2);
%    resLP = add(resLP, resLP, []);
%    baseVars = [baseVars Delta+baseVars];
%    rootVars = [rootVars Delta+rootVars];
% else
%    [resLP baseVars rootVars] = DupPolySOCN(theLP, SizeCone, (N-1));
%    Delta = dims(resLP, 2);
%    resLP = add(resLP, theLP, []);
%    baseVars = [baseVars Delta+(2:SizeCone+1)];
%    rootVars = [rootVars Delta+1];
% end

function apxLP = PolySOCLP(theLP, coneInfo, Epsilon, printLevel)
%PolySOCLP   Computes a polyhedral approximation of a second-ordre cone program.
%            apxLP = PolySOCLP(theLP, coneInfo, Epsilon, printLevel) computes a
%            polyhedral approximation with accuracy Epsilon of the second-order
%            cone program described by theLP (objective and linear constraints)
%            and coneInfo (list of second-order cones).
%            Optional parameter printLevel = 0 => no output
%                                            1 => outputs a summary
%                                            2 => info for each cone (default)
if nargin < 4
   printLevel = 2;
end if printLevel
   disp(sprintf(['Approximating with %4.2g epsilon SOCP with %d cones, ' ...
                 '%d variables and %d constraints.'], Epsilon, ...
                 size(coneInfo, 2), dims(theLP, 2), dims(theLP, 1)));

end
maxEpsilon = inf;
apxLP = theLP;
for indexCone = 1:length(coneInfo)
coneSize(indexCone) = length(coneInfo(indexCone).memb) - 1;
end
[Sorted Order] = sort([coneSize]);
indexCone = 1;
while indexCone <= length(coneInfo)
[coneLP theEpsilon theSteps] = PolySOCN(Sorted(indexCone), Epsilon);
if theEpsilon < maxEpsilon
   maxEpsilon = theEpsilon;
end
nCones = max(find(Sorted == Sorted(indexCone))) - indexCone + 1;
if printLevel >= 2
   disp([sprintf(['-> %d SOC of dimension %d : %g epsilon ' ...
                  'with %d variables, %d constraints ('], nCones, ...
                  Sorted(indexCone), theEpsilon, dims(coneLP, 2), ...
                  dims(coneLP, 1)) mat2str(theSteps) ' steps).']);
end
[NconeLP, baseVars, rootVars] = DupPolySOCN(coneLP, Sorted(indexCone), nCones);
theCones = [coneInfo(Order(indexCone:indexCone+nCones-1)).memb];
reOrder = NaN*ones(1, max([baseVars rootVars]));
reOrder(rootVars) = theCones(1, :);
reOrder(baseVars) = theCones(2:end, :);
apxLP = add(apxLP, NconeLP, reOrder);
indexCone = indexCone + nCones;
end
if printLevel
disp(sprintf(['Final approximation has %4.2g epsilon with %d variables ' ...
              'and %d constraints.'], maxEpsilon, dims(apxLP, 2), ...
              dims(apxLP, 1)));
end

Bibliography

[AA99] E. D. Andersen and K. D. Andersen, The MOSEK interior point optimizer for linear programming: an implementation of the homogeneous algorithm, High Performance Optimization (H. Frenk, C. Roos, T. Terlaky, and S. Zhang, eds.), Applied optimization, vol. 33, Kluwer Academic Publishers, 1999.

[AGMX96] E. D. Andersen, J. Gondzio, Cs. Mészáros, and X. Xu, Implementation of interior-point methods for large scale linear programs, Interior Point Methods of Mathematical Programming (T. Terlaky, ed.), Applied Optimization, vol. 5, Kluwer Academic Publishers, 1996, pp. 189–252.

[Ans90] K. M. Anstreicher, On long step path following and SUMT for linear and quadratic programming, Tech. report, Yale School of Management, Yale University, New Haven, CT, 1990.

[Ans96] K. M. Anstreicher, Potential reduction algorithms, Interior Point Methods of Mathematical Programming (T. Terlaky, ed.), Applied Optimization, vol. 5, Kluwer Academic Publishers, 1996, pp. 125–158.

[ART00] E. D. Andersen, C. Roos, and T. Terlaky, On implementing a primal-dual interior-point method for conic quadratic optimization, in preparation, 2000.

[Bri00] J. Brinkhuis, Communication at the International Symposium on Mathematical Programming, Atlanta, August 2000.

[BTN94] A. Ben-Tal and A. Nemirovski, Potential reduction polynomial-time method for truss topology design, SIAM Journal of Optimization 4 (1994), 596–612.

[BTN98] A. Ben-Tal and A. Nemirovski, On polyhedral approximations of the second-order cone, Tech. report, Minerva Optimization Center, Faculty of Industrial Engineering and Management, Technion - Israel Institute of Technology, Haifa, Israel, 1998, to appear in Mathematics of Operations Research.

[Dan63] G. B. Dantzig, Linear programming and extensions, Princeton University Press, Princeton, N.J., 1963.

[DG00] B. Diricq and Fr. Glineur, Prédire la réussite en première candidature en sciences appliquées : mathématiques ou médiumnité ?, in preparation, 2000.

[Dik67] I. I. Dikin, Iterative solution of problems of linear and quadratic programming, Doklady Akademii Nauk SSSR 174 (1967), 747–748.

[dJRT95] D. den Hertog, F. Jarre, C. Roos, and T. Terlaky, A sufficient condition for selfconcordance with application to some classes of structured convex programming problems, Mathematical Programming, Series B 69 (1995), no. 1, 75–88.

[DPZ67] R. J. Duffin, E. L. Peterson, and C. Zener, Geometric programming, John Wiley & Sons, New York, 1967.

[dRT92] D. den Hertog, C. Roos, and T. Terlaky, On the classical logarithmic barrier method for a class of smooth convex programming problems, Journal of Optimization Theory and Applications 73 (1992), no. 1, 1–25.

[DS97] A. Dax and V. P. Sreedharan, On theorems of the alternative and duality, Journal of Optimization Theory and Applications 94 (1997), no. 3, 561–590.

[ET76] I. Ekeland and R. Temam, Convex analysis and variational problems, Studies in mathematics and its applications, vol. 1, North-Holland publishing company, Amsterdam, Oxford, 1976.

[FM68] A. V. Fiacco and G. P. McCormick, Nonlinear programming: Sequential unconstrained minimization techniques, John Wiley & Sons, New York, 1968, Reprinted in SIAM Classics in Applied Mathematics, SIAM Publications, 1990.

[Fri55] K. R. Frisch, The logarithmic potential method of convex programming, Tech. report, University Institute of Economics, Oslo, Norway, 1955.

[Gli97] Fr. Glineur, Etude des méthodes de point intérieur appliquées à la programmation linéaire et à la programmatiuon semidéfinie, Travail de fin d’études études, Faculté Polytechnique de Mons, Mons, Belgium, June 1997.

[Gli98a] Fr. Glineur, Interior-point methods for linear programming: a guided tour, Belgian Journal of Operations Research, Statistics and Computer Science 38 (1998), no. 1, 3–30.

[Gli98b] Fr. Glineur, Pattern separation via ellipsoids and conic programming, Mémoire de D.E.A., Faculté Polytechnique de Mons, Mons, Belgium, September 1998.

[Gli99] Fr. Glineur, Proving strong duality for geometric optimization using a conic formulation, IMAGE Technical Report 9903, Faculté Polytechnique de Mons, Mons, Belgium, October 1999, to appear in Annals of Operations Research.

[Gli00a] Fr. Glineur, Approximating geometric optimization with l_{p}-norm optimization, IMAGE Technical Report 0008, Faculté Polytechnique de Mons, Mons, Belgium, November 2000, submitted to Operations Research Letters.

[Gli00b] Fr. Glineur, An extended conic formulation for geometric optimization, IMAGE Technical Report 0006, Faculté Polytechnique de Mons, Mons, Belgium, May 2000, submitted to Foundations of Computing and Decision Sciences.

[Gli00c] Fr. Glineur, Polyhedral approximation of the second-order cone: computational experiments, IMAGE Technical Report 0001, Faculté Polytechnique de Mons, Mons, Belgium, January 2000, revised November 2000.

[Gli00d] Fr. Glineur, Self-concordant functions in structured convex optimization, IMAGE Technical Report 0007, Faculté Polytechnique de Mons, Mons, Belgium, October 2000, submitted to European Journal of Operations Research.

[GT56] A. J. Goldman and A. W. Tucker, Theory of linear programming, Linear Equalities and Related Systems (H. W. Kuhn and A. W. Tucker, eds.), Annals of Mathematical Studies, vol. 38, Princeton University Press, Princeton, New Jersey, 1956, pp. 53–97.

[GT00] Fr. Glineur and T. Terlaky, A conic formulation for l_{p}-norm optimization, IMAGE Technical Report 0005, Faculté Polytechnique de Mons, Mons, Belgium, May 2000, submitted to Journal of Optimization Theory and Applications.

[GW95] M. X. Goemans and D. P. Williamson, Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming, Journal of Association for Computing Machinery 42 (1995), no. 6, 1115–1145.

[Hua67] P. Huard, Resolution of mathematical programming with nonlinear constraints by the method of centers, Nonlinear Programming (J. Abadie, ed.), North Holland, Amsterdam, The Netherlands, 1967, pp. 207–219.

[HvM97] T. Terlaky H. van Maaren, Inverse barriers and ces-functions in linear programming, Operations Research Letters 20 (1997), 15–20.

[Jar89] F. Jarre, The method of analytic centers for smooth convex programs, Dissertation, Institut für Angewandte Mathematik und Statistik, Universität Wärzburg, Germany, 1989.

[Jar96] F. Jarre, Interior-point methods for classes of convex programs, Interior Point Methods of Mathematical Programming (T. Terlaky, ed.), Applied Optimization, vol. 5, Kluwer Academic Publishers, 1996, pp. 255–296.

[Kar84] N. K. Karmarkar, A new polynomial-time algorithm for linear programming, Combinatorica 4 (1984), 373–395.

[Kha79] L. G. Khachiyan, A polynomial algorithm in linear programming, Soviet Mathematics Doklady 20 (1979), 191–194.

[Kla74] E. Klafszky, Geometric programming and some applications, Ph.D. thesis, Tanulmányok, No. 8, 1974.

[Kla76] E. Klafszky, Geometric programming, Seminar Notes, no. 11.976, Hungarian Committee for Systems Analysis, Budapest, 1976.

[KM72] V. Klee and G. J. Minty, How good is the simplex algorithm ?, Inequalities, O. Shisha ed., pp. 159–175, Academic Press, New York, 1972.

[LVBL98] M. S. Lobo, L. Vandenberghe, S. Boyd, and H. Lebret, Applications of second-order cone programming, Linear Algebra and its Applications 284 (1998), 193–228.

[Mas93] W. F. Mascarenhas, The affine scaling algorithm fails for λ = 0.999, Tech. report, Universidade Estadual de Campinas, Campinas S. P., Brazil, October 1993.

[Meh92] S. Mehrotra, On the implementation of a primal-dual interior point method, SIAM Journal on Optimization 2 (1992), 575–601.

[MM99] I. Maros and Cs. Mészáros, A repository of convex quadratic programming problems, Optimization Methods and Software 11-12 (1999), 671–681, special issue on interior-point methods (CD supplement with software), guest editors: Florian Potra, Cornelis Roos and Tamás Terlaky.

[Nes96] Y. Nesterov, Nonlinear optimization, Notes from a lecture given at CORE, UCL, Belgium, 1996.

[NN94] Y. E. Nesterov and A. S. Nemirovski, Interior-point polynomial methods in convex programming, SIAM Studies in Applied Mathematics, SIAM Publications, Philadelphia, 1994.

[PE67] E. L. Peterson and J. G. Ecker, Geometric programming: Duality in quadratic programming and l_{p} approximation II, SIAM Journal on Applied Mathematics 13 (1967), 317–340.

[PE70a] E. L. Peterson and J. G. Ecker, Geometric programming: Duality in quadratic programming and l_{p} approximation I, Proceedings of the International Symposium of Mathematical Programming (Princeton, New Jersey) (H. W. Kuhn and A. W. Tucker, eds.), Princeton University Press, 1970.

[PE70b] E. L. Peterson and J. G. Ecker, Geometric programming: Duality in quadratic programming and l_{p} approximation III, Journal on Mathematical Analysis and Applications 29 (1970), 365–383.

[PRT00] J. Peng, C. Roos, and T. Terlaky, Self-regular proximities and new search directions for linear and semidefinite optimization, Technical report, Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada, March 2000, submitted to Mathematical Programming.

[Ren00] J. Renegar, A mathematical view of interior-point methods in convex optimization, to be published by in the MPS/SIAM Series on Optimization, SIAM, New York, 2000.

[Roc70a] R. T. Rockafellar, Convex analysis, Princeton University Press, Princeton, N. J., 1970.

[Roc70b] R. T. Rockafellar, Some convex programs whose duals are linearly constrained, Non-linear Programming (J. B. Rosen, ed.), Academic Press, 1970.

[RT98] C. Roos and T. Terlaky, Nonlinear optimization, Delft University of Technology, The Netherlands, 1998, Course WI387.

[RTV97] C. Roos, T. Terlaky, and J.-Ph. Vial, Theory and algorithms for linear optimization. an interior point approach, Wiley-Interscience Series in Discrete Mathematics and Optimization, John Wiley & Sons, Chichester, UK, 1997.

[Sat75] K. Sato, Production functions and aggregation, North-Holland, Amsterdam, 1975.

[Sch86] A. Schrijver, Theory of linear and integer programming, Wiley-Interscience series in discrete mathematics, John Wiley & sons, 1986.

[Sho70] N. Z. Shor, Utilization of the operation of space dilatation in the minimization of convex functions, Kibernetika 1 (1970), 6–12.

[Stu97] J. F. Sturm, Primal-dual interior-point approach to semidefinite programming, Ph.D. thesis, Erasmus Universiteit Rotterdam, The Netherlands, 1997, published in [Stu99a].

[Stu99a] J. F. Sturm, Duality results, High Performance Optimization (H. Frenk, C. Roos, T. Terlaky, and S. Zhang, eds.), Applied optimization, vol. 33, Kluwer Academic Publishers, 1999, pp. 21–60.

[Stu99b] J. F. Sturm, Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones, Optimization Methods and Software 11-12 (1999), 625–653, special issue on interior-point methods (CD supplement with software), guest editors: Florian Potra, Cornelis Roos and Tamás Terlaky.

[SW70] J. Stoer and Ch. Witzgall, Convexity and optimization in finite dimensions I, Springer Verlag, Berlin, 1970.

[Ter85] T. Terlaky, On l_{p} programming, European Journal of Operations Research 22 (1985), 70–100.

[VB96] L. Vandenberghe and S. Boyd, Semidefinite programming, SIAM Review 38 (1996), 49–95.

[Wri97] S. J. Wright, Primal-dual interior-point methods, SIAM, Society for Industrial and Applied Mathematics, Philadelphia, 1997.

[XY00] G. Xue and Y. Ye, An efficient algorithm for minimizing a sum of p-norms, SIAM Journal on Optimization 10 (2000), no. 2, 551–579.

[Ye97] Y. Ye, Interior point algorithms, theory and analysis, John Wiley & Sons, Chichester, UK, 1997.

[YTM94] Y. Ye, M. J. Todd, and S. Mizuno, An O(\sqrt{n}L)-iteration homogeneous and self-dual linear programming algorithm, Mathematics of Operations Research 19 (1994), 53–67.

Summary

Optimization is a scientific discipline that lies at the boundary between pure and applied mathematics. Indeed, while on the one hand some of its developments involve rather theoretical concepts, its most successful algorithms are on the other hand heavily used by numerous companies to solve scheduling and design problems on a daily basis.

Our research started with the study of the conic formulation for convex optimization problems. This approach was already studied in the seventies but has recently gained a lot of interest due to development of a new class of algorithms called interior-point methods. This setting is able to exploit the two most important characteristics of convexity:

⋄ a very rich duality theory (existence of a dual problem that is strongly related to the primal problem, with a very symmetric formulation),

⋄ the ability to solve these problems efficiently, both from the theoretical (polynomial algorithmic complexity) and practical (implementations allowing the resolution of largescale problems) point of views.

Most of the research in this area involved so-called self-dual cones, where the dual problem has exactly the same structure as the primal: the most famous classes of convex optimization problems (linear optimization, convex quadratic optimization and semidefinite optimization) belong to this category. We brought some contributions in this field:

⋄ a survey of interior-point methods for linear optimization, with an emphasis on the fundamental principles that lie behind the design of these algorithms,

⋄ a computational study of a method of linear approximation of convex quadratic optimization (more precisely, the second-order cone that can be used in the formulation of \underline{216 Summary}

quadratic problems is replaced by a polyhedral approximation whose accuracy that can be guaranteed a priori),

⋄ an application of semidefinite optimization to classification, whose principle consists in separating different classes of patterns using ellipsoids defined in the feature space (this approach was successfully applied to the prediction of student grades).

However, our research focussed on a much less studied category of convex problems which does not rely on self-dual cones, i.e. structured problems whose dual is formulated very differently from the primal. We studied in particular

⋄ geometric optimization, developed in the late sixties, which possesses numerous application in the field of engineering (entropy optimization, used in information theory, also belongs to this class of problems)

⋄ l_{p}-norm optimization, a generalization of linear and convex quadratic optimization, which allows the formulation of constraints built around expressions of the form |ax + b|^{p} (where p is a fixed exponent strictly greater than 1).

For each of these classes of problems, we introduced a new type of convex cone that made their formulation as standard conic problems possible. This allowed us to derive very simplified proofs of the classical duality results pertaining to these problems, notably weak duality (a mere consequence of convexity) and the absence of a duality gap (strong duality property without any constraint qualification, which does not hold in the general convex case). We also uncovered a very surprising result that stipulates that geometric optimization can be viewed as a limit case of l_{p}-norm optimization. Encouraged by the similarities we observed, we developed a general framework that encompasses these two classes of problems and unifies all the previously obtained conic formulations.

We also brought our attention to the design of interior-point methods to solve these problems. The theory of polynomial algorithms for convex optimization developed by Nesterov and Nemirovsky asserts that the main ingredient for these methods is a computable self-concordant barrier function for the corresponding cones. We were able to define such a barrier function in the case of l_{p}-norm optimization (whose parameter, which is the main determining factor in the algorithmic complexity of the method, is proportional to the number of variables in the formulation and independent from p) as well as in the case of the general framework mentioned above.

Finally, we contributed a survey of the self-concordancy property, improving some useful results about the value of the complexity parameter for certain categories of barrier functions and providing some insight on the reason why the most commonly adopted definition for self-concordant functions is the best possible.

About the cover

The drawing depicted on the cover and the variant that is presented on the next page are meant to illustrate some of the topics presented in this thesis, namely the fundamental notions of central path and barrier function for interior-point methods, as well as the existence of multiple types of convex constraints. Each of the small frames represents a convex optimization problem involving two variables (x, y) and the following four constraints:

a. a first linear constraint 5y − 0.9x ≤ 4.5, which defines the upper left boundary of the feasible zone,

b. a second hyperbolic constraint 32xy ≥ 1, which can be modelled as a second-order cone constraint (see Example 3.1 in Chapter 3) and is responsible for the lower left boundary of the feasible region,

c. a third l_{p}-norm constraint |x|^{3/2} + |y|^{3/2} ≤ 0.9 (see Chapter 4), which defines the lower right boundary of the feasible set,

d. and finally a fourth geometric constraint e^{x} + e^{y} ≤ 4.15 (see Chapter 5) to determine the shape of the upper right boundary of the feasible area.

Although they share the same feasible region, the different problems represented in these frames differ by their objective function: each of them has been endowed with a linear objective function pointing towards the direction of the relative position of the frame on the page. For example, the objective functions in the first and second pictures on the cover point towards the north-west and north-north-west directions.

We have drawn for each of these problems some level sets of a suitable barrier function combined with the objective function (more precisely, it is the objective function of problem (CL_{μ}) from Chapter 2 with μ = 1) and the central path corresponding to this barrier function (see again Chapter 2). The endpoints of this central path correspond to the minimum and the maximum of the corresponding objective function on the feasible region.

One can notice that the level sets tend to be shifted in the direction of the objective function, and that the central path can sometimes take surprising turns before reaching its optimal endpoints.

218  About the cover