Masaryk University, Faculty of Science RICCATI MATRIX DIFFERENTIAL EQUATIONS AND STURMIAN THEORY FOR LINEAR HAMILTONIAN SYSTEMS Peter Šepitka Habilitation thesis, 2021 RICCATI MATRIX DIFFERENTIAL EQUATIONS AND STURMIAN THEORY FOR LINEAR HAMILTONIAN SYSTEMS Peter Šepitka Habilitation thesis, 2021 The author is currently an Assistant Professor at the Department of Mathematics and Statistics, Faculty of Science, Masaryk University. He can be reached via the contacts: E-mail: sepitkap@math. muni . cz WWW: https : //www.muni . cz/en/people/175283-peter-sepitka Address: Department of Mathematics and Statistics Faculty of Science, Masaryk University Kotlářská 2, CZ-61137 Brno Czech Republic Phone: +420 - 549497681 Key words and phrases. Riccati matrix differential equation; Linear Hamiltonian system; Controllability; Normality; Comparative index; Conjoined basis; Genus of conjoined bases; Distinguished solution; Proper focal point; Multiplicity of focal point; Sturmian separation theorem; Sturmian comparison theorem; Principal solution; Moore-Penrose pseudoinverse; Orthogonal projector. 2020 Mathematics Subject Classification. 34C10. To my Teacher and Mentor Copyright © 2021 Peter Sepitka Contents Preface ........................................................................................vii 1. Riccati matrix differential equations .....................................................1 1.1. Introduction .............................................................................1 1.2. Genera of conjoined bases ................................................................2 1.3. Riccati matrix differential equations ...................................................... 1.4. Distinguished solutions of Riccati equations .............................................. 1.5. Riccati quotients and comparative index ................................................12 2. Sturmian theory for linear Hamiltonian systems ......................................15 2.1. Review of Sturmian theory for controllable systems......................................15 2.1.1. Sturmian separation theorems....................................................16 2.1.2. Sturmian comparison theorems ...................................................17 2.2. General Sturmian theory on compact interval ...........................................18 2.2.1. Sturmian separation theorems....................................................20 2.2.2. Sturmian comparison theorems ...................................................22 2.3. General Singular Sturmian theory on unbounded intervals ...............................24 2.3.1. Singular Sturmian separation theorems ...........................................25 2.3.2. Singular Sturmian comparison theorems..........................................28 2.3.3. Sturmian comparison theorems for controllable systems ..........................31 2.4. Further research directions ..............................................................33 Declaration about co-authorship ...........................................................35 References ....................................................................................37 A. Paper by Sepitka (JDDE 2020) ......................................................... 41 B. Paper by Sepitka (DCDS 2019) .........................................................61 C. Paper by Sepitka & Simon Hilscher (JDE 2017) .....................................109 D. Paper by Sepitka & Simon Hilscher (JDE 2019) .....................................143 E. Paper by Sepitka & Simon Hilscher (JDE 2020) ..................................... 189 V Preface In this work we study the interrelations between two important concepts from the qualitative theory of differential equations. These are the Riccati differential equation and the separation or comparison properties of zeros of solutions of linear differential systems (called the Sturmian theory). These two concepts are connected by the well known mathematical object - the linear Hamiltonian differential system. This work has been written for obtaining the academic qualification Associate Professor (docent) at the Masaryk University in Brno, Czech Republic. It contains research results achieved by the author in the years 2016-2021 and published in the papers 1. P. Sepitka, Genera of conjoined bases for (non)oscillatory linear Hamiltonian systems: extended theory, Journal of Dynamics and Differential Equations, 32 (2020), no. 3, 1139-1155, 2. P. Sepitka, Riccati equations for linear Hamiltonian systems without controllability condition, Discrete Continuous Dynamical Systems Series A, 39 (2019), no. 4, 1685-1730, 3. P. Sepitka, R. Simon Hilscher, Comparative index and Sturmian theory for linear Hamiltonian systems, Journal of Differential Equations, 262 (2017), no. 2, 914-944, 4. P. Sepitka, R. Simon Hilscher, Singular Sturmian separation theorems on unbounded intervals for linear Hamiltonian systems, Journal of Differential Equations, 266 (2019), no. 11, 7481-7524, 5. P. Sepitka, R. Simon Hilscher, Singular Sturmian comparison theorems for linear Hamiltonian systems, Journal of Differential Equations, 269 (2020), no. 4, 2920-2955. These publications are considered as the main sources, see items [77, 78, 84, 86, 87] in the list of references and Appendices A-E for more details. We also present some additional results in the context of the current literature, which are closely related to those in the above mentioned references, such as in papers [34,85,88,89] and in the monograph [58]. The habilitation thesis is considered as an extended commentary to the published results in the attached papers. We comment on our results in a broader context of the historical literature and the current development of the subject. In Chapter 1 we discuss the Riccati matrix differential equations for possibly uncontrollable linear Hamiltonian systems. We show the variability of these Riccati matrix equations depending on the choice of a genus of conjoined bases, to which the considerations are restricted. The theory of genera of conjoined bases is presented as an introductory part. The results in Chapter 1 are based on the first two papers from the above list. In Chapter 2 we present the Sturmian theory for linear Hamiltonian systems (i.e., the Sturmian separation and comparison theorems), which is based on the properties of the Riccati quotients - symmetric solutions of Riccati matrix differential equations. As a connecting tool we use the comparative index. This object was introduced by J. Elyseeva in 2007 in the connection with discrete oscillations. It was implemented into the continuous time theory by Elyseeva in 2016 and independently by the author and R. Simon Hilscher in 2017 (in the third paper from the above list). By using the comparative index we are able to develop both regular and singular Sturmian theory, including the multiplicities of focal points at infinity. Finally, we wish to mention that along with our investigations in the theory of possibly uncontrollable linear Hamiltonian systems we sometimes derive new results in other fields of mathematics, in particular in linear algebra (matrix analysis, theory of the Moore-Penrose pseudoinverses, and orthogonal projectors) or mathematical analysis (linear control systems). This is documented, for vii Vlil Preface example, in the third section of the first paper mentioned above (Appendix A), in the last section of the fifth paper mentioned above (Appendix E), as well as in our previous papers [79, Appendix 1] and [80, Appendix 1]. We derived these results as needed tools for our investigations, but they may be of independent interest for other researchers. I would like to express gratitude to my collaborator, colleague, friend, and former advisor Roman Simon Hilscher for his continual support, willingness, fruitful discussions, and his advices and comments. Many thanks belong also to the heads of our scientific team, Zuzana Došlá and Petr Hasil. Last, but not least, I thank my family, friends, and especially my Little Sun for their support. Brno, May 2021 Peter Šepitka CHAPTER 1 Riccati matrix differential equations In this chapter we will present the theory of Riccati matrix differential equations associated to linear Hamiltonian differential systems. In particular, we will focus on our recent results on this subject, where we do not impose some traditional assumptions (as we explain below). 1.1. Introduction Let n G N be a given dimension, let X C R be a given interval, and let U : X ->■ R2nx2n be a given piecewise continuous matrix-valued function. Typically we will consider a compact interval X = [a, b] or an unbounded interval X = [a, oo). As a main object of our study we consider the linear Hamiltonian system y' = JH{t)y, tel, (H) where J is the canonical skew-symmetric 2n x 2n matrix. In the n x n block notation we have where A, B, C : X —> Rraxra are piecewise continuous functions such that B(t) and C(t) are symmetric. System (H) then has the equivalent form x = A(t)x + B(t)u, u = C(t)x - AT(t)u, tel. (1.2) With system (H) we associate the Riccati matrix differential equation Q' + Q A(t) + AT(t) Q + Q B(t) Q - C(t) = 0, tel. (R) The connection of the Riccati equation (R) with system (H) is studied in many classical works see e.g. [21,46,58,67-69]. It is known that under the Legendre condition B(t)>0 for all t G [a, oo) (1.3) the Riccati equation (R) has many applications in various disciplines, such as in the oscillation and spectral theory [11,21,58,67-69], filtering and prediction theory [57,68], calculus of variations and optimal control theory [8,12,22,43,47,49,58,63,72,73,102-104], systems theory and control [55,56], exponential dichotomy of perturbed linear Hamiltonian systems [43,56,64], and others (engineering, etc.). We recall that the Riccati matrix differential equation (R) has the distinguished property among the first order differential equations, namely it preserves the ordering of its solutions along the interval [a, oo), see [74,75]. Classical theory of system (H) and equation (R) involves a complete controllability assumption, see e.g. [11,21,46,58,67,69]. This assumption says that vector solutions (x,u) of system (1.1) are not degenerate on X. Specifically, if the function x vanishes on a subinterval of 1q C X, then also u vanishes on Tq, and hence (x,u) = (0,0) by the uniqueness of solutions. This condition is also known as the identical normality of system (H) on X. A characterization of this condition in terms of focal points of conjoined bases of system (H) is presented in Proposition 2.1. When system (H) does not satisfy this complete controllability (identical normality) assumption or when this assumption is not imposed, then we say that system (H) is (possibly) uncontrollable or abnormal. Let us recall several important results in this area. In [65], Reid showed that, under condition (1.3) and when system (H) is completely controllable and nonoscillatory, the Riccati equation (R) has the so-called distinguished solution Q(t) at infinity. More precisely, it is the smallest symmetric solution of (R) existing on an interval [a,oo) for some a > a. In the subsequent paper [66], Reid derived the l 2 Chapter 1. Riccati matrix differential equations existence and the minimality of the distinguished solution of (R) at infinity also for a noncontrollable system(H) by considering invertible principal solutions (X, U) of (H) at infinity. 1.2. Genera of conjoined bases Recently, in [' J-83] the author and Simon Hilscher developed the theory of principal solutions at infinity and antiprincipal solutions at infinity (called also nonprincipal solutions at infinity in some literature) for a general nonoscillatory and possibly abnormal system (H). These notions will be recalled in Section 1.4. They showed the existence of principal and antiprincipal solutions at infinity, whose first component has the rank equal to any integer in the range between n — d^ and n, where the number d^ is the maximal order of abnormality of (H), see below. The above general approach to principal and antiprincipal solutions at infinity naturally requires using the Moore-Penrose pseudoinverse [13,19,59], which in this context substitute the traditionally used invertible matrices. In the above references we also derived a classification of principal and antiprincipal solutions at infinity and their mutual limit properties at infinity. These results are based on the investigation of conjoined bases (X,U) of (H), which have eventually the same image of X(t). The set of all such conjoined bases of (H) is called, according to [80, Definition 6.3], as • a genus of conjoined bases of system (H), and it is denoted by Q. This notion turned out to be a key tool for the study of analytic properties of conjoined bases of system (H), but also for the understanding of the algebraic structure of the set of all conjoined bases of (H). We showed that every genus Q contains some principal solution of (H) at infinity, as well as some antiprincipal solution of (H) at infinity. Moreover, the orthogonal projector representing each genus Q of conjoined bases satisfies a symmetric Riccati matrix differential equation. This result then allowed to obtain an exact description of the structure of the set of all genera of conjoined bases, in particular it forms a complete lattice. The minimal element of this lattice is the so-called minimal genus C/min, which contains all conjoined bases (X, U) with the eventual rank of X(t) equal to smallest possible value n — d^. On the other hand, the maximal element of this lattice is the so-called maximal genus Qmax, which contains all conjoined bases (X, U) with the eventual rank of X(t) equal to largest possible value n, i.e., with X(t) eventually invertible. Note that in the completely controllable case we have d^ = 0, and hence the minimal and maximal genera coincide, i.e., there exists only one single genus Q = C/min = Gmax- Therefore, in the study of abnormal systems (H) we obtain much wider structural variability and the theory of genera of conjoined bases provides a true guideline for potential development of the qualitative theory of these systems. In [78] we extended the theory of genera of conjoined bases to arbitrary systems (H) by removing two key assumptions. Given the unbounded interval X = [a,oo), we consider the case when • the Legendre condition (1.3) is not assumed, and/or • the system (H) may be oscillatory. More precisely, in the new general definition of a genus Q corresponding to a conjoined basis (X, U) of (H), see [78, Definition 4.3], we consider the subspace Iml(t) +Imi?Aoo(t), ie[a,oo), (1.4) where RAoo(t) is the orthogonal projector onto the maximal subspace of eventually degenerate solutions {x = 0,u) of (H) at the point t. It is important to note that, according to [78, Theorem 4.7], every genus Q can be represented by an orthogonal projector Rg(t) satisfying the Riccati type matrix differential equation Rg-A(t)Rg-RgAT(t) + Rg[A(t) + AT(t)}Rg = 0, te[a,oo). (1.5) Also in this much more general case it is possible to show that the set of all genera of conjoined bases of system (H) forms a complete lattice, see [78, Theorem 4.14]. In general, following the standard notation used in [66, Section 3] and [82, Section 2], for a given a G [a, oo) we denote by A[a, oo) the linear space of n-dimensional piecewise continuously difierentiable vector-valued functions u which correspond to the solutions (x = 0,u) of system (H) on [a, oo). The space A[a,oo) is finite-dimensional with d[a, oo) := dimA[a,oo) < n. The number d[a, oo) is called Section 1.2. Genera of conjoined bases 3 the order of abnormality of system (H) on the interval [a,oo). According to [80, Section 6] there exists the limit doo := lim d[t, oo) = max d[t, oo), 0 < d^ < n, (1-6) t->oo te[a,oo) which we call the maximal order of abnormality of (H). Moreover, we define the point «oo := min{« £ [a, oo), d[a, oo) = doo}- Following [78] we consider the orthogonal projector RAoa(t) := Vw±, where Wt := At[«oo, oo), i£[«oo,oo). (1.8) Here the set At[«oo,oo) is the subspace in R™ of the values u(t) of functions u £ A[«oo,oo) at the point t £ [ctoo, oo). The orthogonal projector RAoo(t) plays a crucial role in the new theory of genera of conjoined bases of system (H). In the remaining part of this section we present the main results from [78], see also Appendix A. Definition 1.1 (Genus of conjoined bases). Let (X±, U±) and (A2, U2) be two conjoined bases of (H). We say that (X±,Ui) and (X2,U2) have the same genus (or they belong to the same genus) if there exists a £ [000,00) such that ImXi(i) +ImRAao(t) = ImX2(t) +Imi?Aoo(t), t £ [a, 00), where is defined in (1.7). From Definition 1.1 it follows that there exists a partition of the set of all conjoined bases of (H) into disjoint classes of conjoined bases with the same genus. We will interpret each class Q as a genus itself. The following result provides a fundamental property of conjoined bases of (H) with the same genus. Theorem 1.2. Let (Xi,Ui) and (X.2,U2) be conjoined bases of (H). Then the following statements are equivalent. (i) The conjoined bases (X±, U±) and (X2, U2) belong to the same genus Q. (ii) The equality ImAi(i) +Imi?Aoo(t) = Ini-X^i) +Imi?Aoo(t) holds for every t £ [a^oo). (iii) The equality ImAi(i) +Imi?Aoo(t) = Ini-X^i) +Imi?Aoo(t) holds for some t £ [000,00). Given a genus Q of conjoined bases of (H), the subspace Iml(i) + Im RAoo(t) does not depend on the particular choice of such a conjoined basis (A, U) belonging to Q. Therefore, the orthogonal projector onto ImA(t) +Imi?Aoo(t), i.e., the matrix Rg(t):=VVt, where Vt := ImX(t) + Imi?Aoo(t), te [0^,00), (1.9) is uniquely determined for each genus Q. The following two theorems provide basic properties of orthogonal projectors Rg(t) defined in (1.9). Moreover, they show how to classify a genus Q of conjoined bases of (H) via its associated projector Rg(t) in (1.9). Theorem 1.3. Let Q be a genus of conjoined bases of (H) and let Rg(t) be the orthogonal projector defined in (1.9). Then the matrix Rg(t) is a solution of the Riccati equation (1.5) on [«00,00) and the inclusion Imi?Aoo(t) C ImRg(t) holds for every t £ [a^, 00). Theorem 1.4. Let a £ [«00,00) be fixed and let R £ M.nxn be an orthogonal projector satisfying ImflAm(a) C Imi?. Then there exists a unique genus Q of conjoined bases of (H) such that its corresponding orthogonal projector Rg(t) in (1.9) satisfies Rg(a) = R. We note that for every genus Q its associated orthogonal projector Rg(t) in (1.9), as a solution of (1.5) on [«00,00), has constant rank on the whole interval [«oo,°°), i-e-> rg := rankRg(t), ££[«007°°). (1-10) In this context, we may adopt for the number rg the terminology rank of the genus Q and write rankC/ := rg, compare also with [81, Remark 6.4]. In particular, we have n — doo < rankC/ < n. (1-H) 4 Chapter 1. Riccati matrix differential equations Let us denote by the symbol T the set of all genera of conjoined bases of (H). In the next definition we introduce an ordering on the set of all genera of conjoined bases of (H) in terms of their corresponding orthogonal projectors in (1.9). Definition 1.5. Let Q and ~H be two genera of conjoined bases of (H) and let Rg(t) and Rn(t) be their corresponding orthogonal projectors in (1.9), respectively. We say that the genus Q is below the genus ~H (or that the genus ~H is above the genus Q) and we write Q ■< ~H if the inclusion Im Rg{t) C Im RH(t) holds for all t £ [0^,00). Theorem 1.6. The relation ^ from Definition 1.5 is an ordering on the set T. We note that the genera Q and ~H satisfy Q ■< ~H if and only if the inclusion Im Rg{a) C Im RH(a) holds for some a G ^,00). In particular, if the orthogonal projector Rg(t) satisfies Rg(t) = RAoo(t) on [qqo, 00), then the genus Q = C/min is called minimal, while if Rg(t) = I on 00), then the genus Q = 5mm is called maximal. Theorem 1.7. The ordered set (T, ^) is a complete lattice. In particular, the minimal genus C/min is the least element of T with respect to the ordering ^, while the maximal genus C/max is the greatest element ofT with respect to -<. We can describe explicitly the infimum Q AH and the supremum Q V ~H of two genera Q and 7-L of the set r. More precisely, if Rg(t) and Rn(t) are the orthogonal projectors associated to the genera Q and ~H, then Q A"H is the genus of conjoined bases corresponding to the orthogonal projector onto the subspace ImRg(t) DimRH(t) on ^,00), and QVH is the genus of conjoined bases corresponding to the orthogonal projector onto the subspace ImRg(t) + ImRH(t) on [0«,, 00). The next theorem characterizes the conjoined bases of (H) belonging to the minimal genus C/min-We also show that the principal solution of (H) at the point a G [ctoo, 00) belongs to the minimal genus £min- The principal solution of (H) at the point a is defined as the matrix solution of (H) satisfying the initial condition Xa(a) = 0, Ua{a) = /, (1.12) Theorem 1.8. Let (X, U) be a conjoined basis of (H). Then (X, U) belongs to the minimal genus C/min if and only if the inclusion ImX(t) C ImRAoo(t) holds for some (and hence for every) t G ^,00). In particular, for every a > the principal solution (Xa, Ua) at the point a belongs to C/min- In the final theorem of this section we provide important properties of nonoscillatory conjoined bases from a given genus Q. Theorem 1.9. Let Q be a genus of conjoined basis of (H) with the corresponding orthogonal projector Rg(t) in (1.9). Moreover, let (X,U) be a conjoined basis of (H) with constant kernel on [a, 00) C [qocOo) such that (X,U) belongs to Q and let R(t) be the orthogonal projector onto the subspace ImX(i) for every t G [a, 00). Then the equality Rg(t) = R(t) holds for all t G [a, 00). In the final paragraph of this section we comment on the connection of the above results with those in [80], where it is assumed that the Legendre condition (1.3) holds and that system (H) is nonoscillatory. In particular, for every conjoined basis (X,U) of (H) there exists a G [000,00) such that (X, U) has constant kernel on [a, 00). Moreover, let Q be the genus of conjoined bases such that (X, U) G Q and let Rg(t) be its corresponding orthogonal projector in (1.9). Then the rank rg of Q defined in (1.10) coincides with the rank r of any conjoined basis (X,U) of the genus Q. In view of (1.11) we then obtain that n ~ doo < rankX(i) < n, £ £ [a, 00), (1-13) for every conjoined basis (X, U) of system (H) with constant kernel on the interval [a, 00). Moreover, by Theorem 1.9 we have that ImX(t) = ImRg(t) on [a,00). Therefore, from (1.9) and Definition 1.1 it follows that two conjoined bases (X±, U±) and (X2, U2) of (H) belong to the same genus if and only if there exists a £ [aco,oo) such that the equality Im Ji(i) = Ini-X^i) hold for every t £ [a, 00). This observation shows that the concept given in Definition 1.1 generalizes the definition of genus of conjoined bases introduced in [80, Definition 6.3] for a nonoscillatory system (H). We also note that Section 1.3. Riccati equations 5 the result about the structure of the set of all genera of conjoined bases presented in Theorems 1.6 and 1.7 are in full agreement with the corresponding results in [82, Section 4]. Finally, the result in Theorem 1.8 generalizes [82, Proposition 4.7] to a possibly oscillatory system (H). 1.3. Riccati matrix differential equations The study of Riccati matrix differential equations associated with uncontrollable linear Hamilton-ian systems is also motivated by several situations in the literature. For example, in [73, pg. 886], [8, pp. 621-622], [48, Sections 4 and 6], and [49, pp. 17-18] the authors use a cascade system of three differential equations for the investigation of calculus of variations or optimal control problems with variable endpoints - the Riccati equation (R), a linear differential equation, and an integrator. These three differential equations are together equivalent to a Riccati equation in dimension 2n, which corresponds to an uncontrollable system (H) in dimension 4n. This connection is discussed in details in [48, Remark 6.3]. Among other situations we also mention the occurrence of the symmetric solutions of the implicit Riccati equation Rg(t)[Q'+ QA(t) + AT(t)Q + QB(t)Q-C(t)]Rg(t) = 0, tel, (1.14) in the study of nonnegative quadratic functional associated with possibly an uncontrollable system (H), see [50, Section 6]. The above mentioned extended theory of genera of conjoined bases of system (H) based on the subspaces Im Rg(t) in (1.9) points to new possibilities how to deal with the Riccati matrix differential equations in the context of abnormal linear Hamiltonian systems (H). In particular, it shows how to implement in a proper way the theory of the Riccati type differential equations (R) or (1.14) into the theory of linear Hamiltonian systems (H). Consider the unbounded interval X = [a,oo). The presented approach is novel in three aspects. Namely, • we do not require any controllability assumption on system (H), • for every genus Q we associate a Riccati equation Q' + QA(t)+AT(t)Q + QB(t)Q-C(t) = 0, te [0^,00), (R) where the coefficients A(t), B(t), and C(t) are given by A(t) :=A(t)RS(t)-AT(t)[I-Rg(t)], 1 B(t) :=B(t), \ te [aoo.oo), (1.15) C(t) :=Rg(t)C(t)Rg(t), J with the corresponding orthogonal projector Rg(t) defined in (1.9), • we show that every such a Riccati equation (TV) possesses a distinguished solution at infinity (defined in a suitable way), which corresponds to a principal solution of (H) at infinity from the genus Q. Given a genus Q of conjoined bases of (H), we show a fundamental connection between the symmetric solutions Q(t) of (JZ) on [a, oo) with some a > satisfying ImQ(i) C lmRg(t), ie[a,oo), (1.16) and the conjoined bases (X, U) of (H) with constant kernel on [a, oo), which belong to Q. This allows us to define in a proper way a distinguished solution Q(t) at infinity for each Riccati equation (TZ), which corresponds to a principal solution (X,U) of (H) at infinity in Q. Then for every symmetric solution Q(t) of (TV) on [a, oo) with (1.16) there exists a distinguished solution Q(t) of (TV) satisfying the inequality Q(t)>Q(t) on[a,oo). (1.17) The above results are particularly important for the minimal genus Q = C/min, which is formed by the conjoined bases (X, U) of (H) with minimal possible rank of the matrix X(t), i.e., with rankX(t) = n — doo on [a, oo). In this case the associated distinguished solution Qmin(t) at infinity is unique and minimal among all symmetric solutions Q(t) of (TV) satisfying (1.16). This latter situation generalizes the classical controllable results of Reid and Coppel [21,65,67], since in this case = 0 and the 6 Chapter 1. Riccati matrix differential equations orthogonal projector Rg(t) = I on [a, oo), so that the Riccati equation (JZ) reduces to (R). We note that the original results by Reid [66, 68] for noncontrollable system (H) and Riccati equation (R) correspond in our new theory to the maximal genus Q = C/max of conjoined bases (X, U) with eventually invertible matrix X(t), i.e., to Rg(t) = I on [a, oo). Therefore, the present study can be regarded as a generalization and completion of the theory of the Riccati equations (R) for completely controllable systems (H) using the minimal genus Q = Qm{n, as well as the noncontrollable systems (H) using the maximal genus Q = Qmax- In the remaining part of this section we present the main results from [77, Sections 4-6], see also Appendix B. In the first set of results (Theorems 1.10-1.13) we describe basic properties of solutions of the Riccati equation (JZ). Theorem 1.10. Let Q be a genus of conjoined bases of (H) with the corresponding orthogonal projector Rg(t) in (1.9) and let Q(t) be a solution of the Riccati equation (JZ) on [a, oo) C [a^, oo) with in (1.7). Then also the matrices Rg(t) Q(t), Q(t) Rg(t), and Rg(t) Q(t) Rg(t) solve (JZ) on [a, oo). Theorem 1.11. With the assumptions and notations of Theorem 1.10, the matrix Q(t) satisfies the inclusion ImQ(t) C ImRg(t), resp. the inclusion ImQT(t) C Im Rg(t), for all t G [a,oo) if and only if the inclusion ImQ(to) C ImRg(to), resp. the inclusion Im QT(in) Q Im-Rg(£n)> holds for some point to G [a, oo). Theorem 1.12. Let Q be a genus of conjoined bases of (H) with the corresponding matrix Rg(t) in (1.9) and let Q(t) and Q(t) be symmetric solutions of the Riccati equation (JZ) on [a, oo) C [0^,00). Then the quantities rank [Q(t) — Q(t)] and ind [Q(t) — Q(t)] are constant on [a, 00). (1-18) In particular, the inequality Q(t) > Q(t) holds on [a, 00) if and only if Q(a) > Q(a), and the inequality Q(t) > Q(t) holds on [a, 00) if and only if Q(a) > Q(a). Theorem 1.13. Assume (1.3) and let Q be a genus of conjoined bases of (H) with the corresponding matrix Rg(t) in (1.9). Let Q(t) be a symmetric solution of the Riccati equation (JZ) on [a, 00) C [«00, 00) and let Q(t) be a symmetric solution of (JZ) satisfying the initial condition Q(a) > Q(a). Then the matrix Q(t) solves (JZ) on the whole interval [a, 00) such that the inequality Q(t) > Q(t) holds for all t G [a, 00). Theorem 1.14. Let Q be a genus of conjoined bases of (H) and let Rg(t) be its corresponding matrix in (1.9). Moreover, let Q(t) be a symmetric solution of (JZ) on [a, 00) C [a^oo) satisfying condition (1.16). Let j3 G [a, 00) and K G M.nxn be given and consider the solution Q(t) of (JZ) with Q(j3) = K. Then the following statements are equivalent. (i) The matrix Q(t) solves the Riccati equation (JZ) on the whole interval [a, 00) such that Rg(t) Q(t) Rg(t) = Q(t) holds for every t G [a, 00). (ii) The matrix K satisfies the equality Rg(j3) KRg(j3) = Q(j3). Let Q be a genus of conjoined bases of (H) and let Rg(t) be its representing orthogonal projector in (1.9). For a given solution Q(t) of the Riccati equation (JZ) on a subinterval [a, 00) C [a^, 00) we consider the following system of first order linear differential equations 0' = [A(t)+B(t)Q(t)\e, Q' = A(t) n + [I- Rg(t)]{C(t) - [A(t) + AT(t)] Q(t)} 0, together with the initial conditions Q(a)=K, tt(a) = L, (1.20) where the matrices K,L G Mraxra satisfy ImK C lmRg(a), ImL C Ker Rg(a), rank (KT, LT)T = n. (1.21) t G [a, 00) (1.19) Section 1.3. Riccati equations 7 The initial value problem (1.19)-(1.20) serves for the formulation of the main results of this section. The first equation in (1.19) is motivated by the approach in [68, Chapter 2, Lemma 2.1], which is adopted here to the setting of uncontrollable systems (H). According to [77, Remark 4.8 and Proposition 4.9] initial value problem (1.19)-(1.20) with (1.21) has always the solution (0, Q), which is unique up to a right nonsingular constant multiple. Moreover, the matrix has a constant kernel on [a,oo) and ImO(t) = ImRg(t), Imfi(i) = Kei Rg(t), rank (0T(t), ^T(t))T = n, ie[a,oo). (1.22) These properties of the matrix ©(£) allow us to define the function Fa(t):= f oo exists, where the matrix Da is symmetric and nonnegative definite with ImDQ C ImRg(a). The next two results extend the well known correspondence between the symmetric solutions Q(t) of the classical Riccati equation (R) on [a, oo) and conjoined bases (A, U) of (H) with X(t) invertible on [a, oo), i.e., Q(t) = U(t)X~1(t) on[a,oo) (1.25) to the case of possibly noninvertible X(t) on [a, oo). For this purpose we utilize the Moore-Penrose generalized inverse X^(t) of the matrix X(t), called also the pseudoinverse, see e.g. [13], [14, Chapter 6], and [19, Section 1.4]. Given a conjoined basis (A, U) of (H) and a point a G [a, oo), the matrix Q(t) defined by Q(t) := X(t) X\t)U(t) X\t) = R(t)U(t) X\t), te[a,oo), (1.26) is called the Riccati quotient associated with the conjoined basis (X,U) on the interval [a,oo). Here R(t) := X(t)X^(t) is the orthogonal projector onto the subspace ImA(t) for all t G [a,oo). By [71, pg. 24] the matrix Q(t) is symmetric and satisfies on [a, oo) the properties XT{t)Q{t) X(t) = XT(t)U(t), Im Q(i) Clmfl(i), Q(t) X(t) = R(t)U(t). (1.27) In addition, if (X,U) has constant kernel on [a, oo), then by [19, Theorems 10.5.1 and 10.5.3] the matrix X\t) is piecewise continuously differentiable on [a, oo), and hence also the matrix Q(t) is piecewise continuously differentiable on [a, oo). Note that when X(t) is an invertible matrix, then the Riccati quotient Q(t) in (1.26) reduces to (1.25). Theorem 1.15. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (1.9). Moreover, let (X,U) be a conjoined basis of (H) belonging to Q such that (X,U) has constant kernel on a subinterval [a, oo) C [a^, oo) and let Q(t) be the corresponding Riccati quotient in (1.26). Then the matrix Q(t) is a symmetric solution of the Riccati equation (JZ) on [a,oo) such that the condition in (1.16) holds and the matrices ©(£) and Q(t) defined by G(t):=X(t), n(t):=U(t)-Q(t)X(t), te[a,oo), (1.28) solve the initial value problem (1.19)-(1.20) on [a,oo) with (1.21). Theorem 1.16. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (1.9) and let Q(t) be a solution of the Riccati equation (JZ) on [a, oo) C [aoo,oo) such that the matrix Rg(t) Q(t) Rg(t) is symmetric on [a, oo). Moreover, let(Q,Q) be a solution of (1.19)-(1.20) on [a, oo) with (1.21) and define the matrices X(t):=G(t), U(t):=Q(t)G(t)+n(t), te[a,oo). (1.29) 8 Chapter 1. Riccati matrix differential equations Then the following statements hold. (i) The pair (X, U) is a conjoined basis of (H) such that (X, U) has a constant kernel on [a, oo) and belongs to the genus Q. (ii) The matrix Rg(t) Q(t) Rg(t) is the Riccati quotient in (1.26) associated with the conjoined basis (X, U) on [a, oo ), i.e., the equality Rg(t) Q(t) Rg(t) = R(t) U(t) X^ (t) holds for all t G [a, oo), where R(t) is the corresponding orthogonal projector onto ImX(t). Let Q be a genus of conjoined basis of (H) with the associated matrix Rg(t) in (1.9) and let [a, oo) C [qqo, oo) be a given interval. The results in Theorems 1.15 and 1.16 provide a correspondence between the set of all conjoined basis (X, U) of (H) with constant kernel on [a, oo), which belong to the genus Q, and the set of all symmetric solutions Q(t) of the Riccati equation (TV) on [a, oo) satisfying condition (1.16). More precisely, for every such a conjoined basis (X,U) its Riccati quotient Q(t) in (1.26) is a symmetric solution of (TV) on [a, oo) with (1.16). Conversely, if Q(t) is a symmetric solution of (7Z) on [a,oo) satisfying (1.16), then there exists a conjoined basis (X, U) of (H) from the genus Q with constant kernel on [a, oo) such that Q(t) is its corresponding Riccati quotient from (1.26). The last part of this section is devoted to the implicit Riccati equations (1.14) and Rg(t)[Q' + QA(t)+AT(t)Q + QB(t)Q-C(t)]Rg(t) = 0, [a,oo), (1.30) with a G [«„0,00). These implicit Riccati equations were used in [50, Section 6] in several criteria characterizing the nonnegativity and positivity of the associated quadratic functional. Theorem 1.17. Let Q be a genus of conjoined bases of (H) and let Rg(t) be the corresponding orthogonal projector in (1.9). Moreover, let Q(t) be a symmetric matrix defined on [a, 00) C [a^oo) such that condition (1.16) holds. Then the following statements are equivalent. (i) The matrix Q(t) solves the Riccati equation (TV) on [a, 00). (ii) The matrix Q(t) solves the implicit Riccati equation (1.30) on [a, 00). (iii) The matrix Q(t) solves the implicit Riccati equation (1.14) on [a, 00). The above result shows that under a certain assumption we can transfer the problem of solving the implicit Riccati matrix differential equations (1.30) and (1.14) into a problem of solving the explicit Riccati matrix differential equation (JZ). 1.4. Distinguished solutions of Riccati equations In this section we study, for a given genus Q, symmetric solutions of the Riccati equation (7V), which correspond to principal solutions of (H) at infinity belonging to the genus Q (to be defined below). This correspondence is based on the results in Theorems 1.15 and 1.16. In particular, we establish the results about distinguished solutions of (TV) at infinity regarding their relationship to principal solutions at infinity and to the nonoscillation of system (H), their interval of existence, their mutual classification within the genus Q, and their minimality in a suitable sense. It may be surprising that these results comply with the known theory of distinguished solutions of the Riccati equation (R) for a controllable system (H) only partially. In many aspects the presented theory for general uncontrollable system (H) is substantially different. This is related to the nature of the problem, since for each genus Q of conjoined bases of (H) there is a different Riccati equation (TV), but even within one genus Q there may be many distinguished solutions of (TV) at infinity. We discuss these issues in Remark 1.31 at the end of this section. We note that the true uniqueness and minimality of the distinguished solution of (TV) at infinity is satisfied only in the minimal genus č/min (see Theorem 1.30). The following definition extends the notion of a distinguished solution (also called a principal solution) of (R) at infinity for a controllable system (H) in [ , pg. 53]. Definition 1.18. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (1.9). A symmetric solution Q(t) of the Riccati equation (TV) is said to be a distinguished solution at infinity if the matrix Q(t) is defined on an interval [a, 00) C [a^oo) and its corresponding matrix Fa(t) in (1.23) satisfies F^(t) ->0asi->oo. Section 1.4. Distinguished solutions 9 The notion in Definition 1.18 also extends the distinguished solution of the Riccati equation (R) introduced by W. T. Reid in [66, Section IV] and [68, Section 2.7], which in our context corresponds to the maximal genus Q = C/max (f°r which Rg(t) = I). The main results of this section compare the properties of distinguished solutions at infinity of the Riccati equation (JZ) with the properties of principal solutions of system (H) at infinity belonging to the genus Q. For this purpose we recall the definition of the latter object. Following [80, Definition 7.1], we say that a conjoined basis (X,U) of (H) is a principal solution at infinity if (X,U) has constant kernel on [a,oo) and its corresponding matrix Sa(t) defined by Sa(t) := f X\s)B(s)X^T(s)ds, te[a,oo), (1.31) J a satisfies Sa(t) —> 0 as t —> oo. In this case we will say that (X,U) is a principal solution of (H) at infinity with respect to the interval [a, oo). By (1.13), the principal solutions of (H) can be classified according to the rank of X(t) on [a, oo). In particular, the minimal principal solution (^min; C^min) °f (H) at infinity satisfies rankXmin(i) = n — d^, while the maximal principal solution (Xmax,Umax) of (H) at infinity is determined by rankXmax(t) = n, hence Xmax(t) is invertible on [a, oo), see [80, Remark 7.2]. In the next proposition we recall from [80, Theorem 7.6] and [79, Theorems 7.6] the characterization of the nonoscillation of system (H) by the existence of a principal solution of (H) at infinity with any possible rank, as well as the uniqueness of the minimal principal solution. Proposition 1.19. Assume that (1.3) holds. Then the following statements are equivalent. (i) System (H) is nonosdilatory. (ii) There exists a principal solution of (H) at infinity. (iii) For any integer r satisfying n — d^ < r < n there exists a principal solution of (H) at infinity with rank equal to r. In particular, system (H) is nonos dilatory if and only if there exists a minimal principal solution of (H) at infinity. In this case the minimal principal solution is unique up to a right nonsingular constant multiple. In [80, Equation 7.4] we defined for a nonoscillatory system (H) the point amin S [a,oo) by cimin := inf {a £ [a7 oo), (Amin, ?7min) has constant kernel on[a,oo)}, (1.32) where (Amin, Umin) is the minimal principal solution of (H) at infinity. We note that the equality d[a, oo) = doo holds for every a > <5min, see [80, Theorem 7.9]. In turn, combining this fact with formula (1.7) we obtain that ^[^min; ^oo; i.e., Amin ^ ^oo* In the remaining part of this section we present the main results from [77, Sections 3 and 7], see Appendix B. The following two results show that in the context of Theorems 1.15 and 1.16 the distinguished solutions of the Riccati equation (TV) correspond to the principal solutions of (H) at infinity from the genus Q. Theorem 1.20. Let Q be a genus of conjoined bases of (H) and Rg(t) be the orthogonal projector in (1.9). Moreover, let Q(t) be a distinguished solution of (JZ) at infinity with respect to the interval [a, oo) C [aocoo). Then every conjoined basis (X,U) of (H), which is associated with Q(t) on [a, oo) via Theorem 1.16, is a principal solution of (H) at infinity with respect to [a, oo) belonging to Q. Theorem 1.21. Let (X,U) be a principal solution of (H) at infinity with respect to the interval [a, oo), which belongs to a genus Q. Moreover, let Q(t) be the Riccati quotient in (1.26) associated with (X,U) on [a, oo). Then Q(t) is a distinguished solution of the Riccati equation (JZ) at infinity with respect to [a, oo). 10 Chapter 1. Riccati matrix differential equations From Theorems 1.20 and 1.21 it follows that the property of the existence of a principal solution of (H) at infinity in the genus Q, as stated in [80, Theorem 7.12], transfers naturally to the existence of a distinguished solution of the associated Riccati equation (TV). Corollary 1.22. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (1.9). Then there exists a principal solution of (H) at infinity belonging to the genus Q if and only if there exists a distinguished solution of the Riccati equation (JZ) at infinity. In this case, the set of all Riccati quotients in (1.26), which correspond to the principal solutions (X,U) of (H) at infinity from the genus Q, coincides with the set of all matrices Rg Q Rg, where Q is a distinguished solution of (JZ) at infinity. In the following result we characterize the nonoscillation of system (H) in terms of the existence of a distinguished solution of (JZ) in a given (or every) genus Q. This corresponds to Proposition 1.19 regarding the principal solutions of (H) at infinity. Theorem 1.23. Assume that (1.3) holds. Then the following statements are equivalent. (i) System (H) is nonosdilatory. (ii) There exists a distinguished solution of the Riccati equation (JZ) for some genus Q. (iii) There exists a distinguished solution of the Riccati equation (JZ) for every genus Q. The result in Theorem 1.23 justifies the development of the theory of genera of conjoined bases for possibly oscillatory system (H) in Section 1.2. Of course, assuming that system (H) is nonoscillatory, then it is sufficient to use the theory of genera of conjoined bases from [80, Section 6] and [82, Section 4] for the construction of distinguished solutions of the Riccati equation (JZ) for a genus Q. It is the converse to this implication, which requires a more general approach, since in this case we need to define the coefficients of equation (1Z) without the assumption of nonoscillation of system (H). In the following result we present a mutual classification of all distinguished solutions of (JZ). This classification is formulated in terms of the initial values of the involved distinguished solutions at some point a from the maximal interval (amin; oo). Theorem 1.24. Assume that (1.3) holds and system (H) is nonoscillatory with <5min and R^if) defined in (1.32) and (1.8), respectively. Let Q be a genus of conjoined bases of (H) and let Rg(t) be the matrix in (1.9). Moreover, let Q(t) be a distinguished solution of the Riccati equation (JZ) at infinity. Then a symmetric solution Q(t) of (JZ) defined on a neighborhood of some point a £ (amin, oo) is a distinguished solution at infinity if and only if ^Aoo(a) Q(ot) RAco(a) = RAco(a) Q(a) RAco(a). (1.33) In the next three results we study the minimality of distinguished solutions of (JZ). This minimality property needs to be understood in the following sense. For every symmetric solution Q(t) of (JZ) there exists a distinguished solution of (JZ), which exists on the same interval and is at the same time smaller than Q(t) on this interval (Theorems 1.25 and 1.26). On the other hand, any symmetric solution of (H), which is smaller than a distinguished solution of (H) on some interval, is a distinguished solution itself with respect to this interval (Theorem 1.27). However, in general there is no universal "smallest" distinguished solution of (JZ), see Remark 1.28 below. We note that in the first result we consider the case when the solutions satisfy condition (1.16), while in the second and third result this assumption is removed. Theorem 1.25. Assume (1.3). Let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (1.9) and let Q(t) be a symmetric solution of the Riccati equation (JZ) on [a, oo) C [a^, oo) such that inclusion (1.16) holds. Then there exists a distinguished solution Q(t) of (JZ) at infinity with respect to [a, oo) satisfying(1.16) such that Q(t) > Q(t) for every t £ [a,oo). Theorem 1.26. Assume (1.3). Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (1.9). Let Q(t) be a symmetric solution of the Riccati equation (JZ) on [a, oo) C [«□0, oo). Then there exists a distinguished solution Q(t) of (JZ) at infinity with respect to [a, oo) such that the inequality Q(t) > Q(t) holds for every t £ [a, oo). Section 1.4. Distinguished solutions 11 We note that the converse to Theorem 1.26 also holds. More precisely, if Q(t) is a distinguished solution of (JZ) at infinity with respect to the interval [a,oo), then every symmetric solution Q(t) of (JZ), which satisfies the condition Q(a) > Q(a), exists on the whole interval [a,oo) and the inequality Q(t) > Q(t) holds for every t G [a, oo). This observation is a direct application of Theorem 1.13 with Q := Q and Q := Q. Theorem 1.27. Assume (1.3) and let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (1.9). Let Q(t) be a distinguished solution of the Riccati equation (JZ) with respect to the interval [a, oo) C [aocoo). Moreover, let Q(t) be a symmetric solution of (JZ) on [a,oo) satisfying the initial condition Q(a) > Q(a). Then Q(t) is a distinguished solution of (JZ) at infinity with respect to [a, oo) and the inequality Q(t) > Q(t) holds for all t G [a, oo). Remark 1.28. Given a genus Q of conjoined bases of (H) with the matrix Rg(t) defined in (1.9), let Q(t) be a distinguished solution of (JZ) at infinity with respect to the interval [a, oo) C [a^oo). Then there exist distinguished solutions Q*(i) and Q**(t) of (JZ) satisfying &(*) < Q(t) < &*(*), t G [a,oo). (1.34) The solutions Q*(t) and Q**(i) are given, for example, by the initial conditions Q*{ot) = Q(o) - I + RAoo(a) and Q««(a) = Q(a) + I - RAoo(a), (1.35) where RAco(t) is the orthogonal projector defined in (1.8). Therefore, for the case of a general (not necessarily controllable) system (H) the partially ordered set of all distinguished solutions of (JZ) has neither a minimal element nor a maximal element. The considerations in Theorems 1.24 and 1.25 show that for the minimal genus C/min, i.e., for Rg(t) = RAco(t), there exists a uniquely determined distinguished solution of (JZ) with A(t) := A(t)RAoo(t) - AT(t)[I - RAoo(t)}, 1 B(t) :=B(t), \ t G [«00,00), (1.36) C(t) :=RAoo(t)C(t)RAoo(t), J which is the smallest element in the set of all symmetric solutions Q(t) of (JZ) satisfying (1.16). Definition 1.29. Let C/min be the minimal genus of conjoined bases of (H) with the minimal orthogonal projector RAoo(t) in (1.8). A symmetric solution Q(t) of the Riccati equation (JZ) with the coefficients in (1.36) is said to be a minimal distinguished solution at infinity if the matrix Q(t) is defined on an interval [a, oo) C [aoo,oo) such that ImQ(i) C Imi?Aoo(t), ie[a,oo), (1.37) and its corresponding matrix Fa(t) in (1.23) satisfies Fa(t) —> 0 as t —> oo. The following result shows the existence and uniqueness of the minimal distinguished solution of (JZ) for the minimal genus C/min, as well as its minimality property. Theorem 1.30. Assume (1.3). Then system (H) is nonosdilatory if and only if there exists a minimal distinguished solution Q(t) of the Riccati equation (JZ) with the coefficients in (1.36). In this case, the minimal distinguished solution Q(t) is determined uniquely and any symmetric solution Q(t) of (JZ) on [a, oo) C [a^, oo) with (1.37) satisfies Q(t) > Q(t) on [a, oo). The minimal distinguished solution of (JZ) at infinity in Theorem 1.30 will be denoted by <2mm-The minimal distinguished solution <2mm plays for the theory of Riccati differential equations (JZ) or (R) a similar role as the minimal principal solution (Amin,?7min) of (H) at infinity for the theory of principal solutions at infinity. Remark 1.31. When system (H) is completely controllable, the main results of this section give the classical statements about the distinguished solutions at infinity of the Riccati equation (R). More precisely, the following holds. 12 Chapter 1. Riccati matrix differential equations • The results in Corollary 1.22 and Theorem 1.24 yield the correspondence between the unique principal solution of (H) at infinity and the unique distinguished solution of (R) at infinity, see [21, pg. 53] or [68, pp. 45-46]. • The result in Theorem 1.23 provides a characterization of the nonoscillation of system (H) in terms of the existence of the unique distinguished solution of (R) at infinity, see the necessary condition in [67, Theorem VII.3.3]. Note that the nonoscillation of (H) is defined in [67, Section VII.3] in terms of disconjugacy of (H), i.e., in terms of the nonexistence of mutually conjugate points, which is a stronger concept than the nonoscillation of (H). We note also that the sufficiency part of Theorem 1.23 is new also in the completely controllable case. • The results in Theorems 1.26 and 1.27 yield the minimality property of the unique distinguished solution of (R) at infinity, see [21, Theorem 8, pg. 54] or [68, Theorem IV.4.2]. Indeed, in this case d^ = 0 and there is only one minimal/maximal genus of conjoined bases of (H). This implies that = a and the orthogonal projector RAoo(t) in (1.8) satisfies RAoo(t) = I on [a, oo). Therefore, the unique Riccati equation (JZ) associated with the minimal/maximal genus coincides with the classical Riccati equation (R). Moreover, under the Legendre condition (1.3) the nonoscillation of system (H) is then equivalent with the existence of a unique (minimal) distinguished solution Q of (R) at infinity. In addition, the matrix Q constitutes the smallest symmetric solution of the Riccati equation (R), that is, every symmetric solution Q of (R) on [a, oo) C [a, oo) satisfies inequality (1.17). 1.5. Riccati quotients and comparative index The Riccati quotient Q(t) defined in (1.26), resp. in (1.25), represents an important tool in the investigations related to the Sturmian theory of system (H). In particular, the results in Section 2.1 show that the changes in the index (i.e., the number of negative eigenvalues) of the difference of two such Riccati quotients determine the difference between the numbers of focal points of two conjoined bases of a completely controllable system (H). A mathematical tool, which describes in full generality such behavior, is the comparative index invented by Elyseeva [32,33]. The comparative index fi(Y, Y) and the dual comparative index fJ-*(Y, Y) of two constant real 2n x n matrices Y = (X, U) and Y = (X, U) satisfying YTJY = 0, YTJY = 0, rankF = n = rankf (1.38) are nonnegative integers (between 0 and n) defined by the equations n(Y, Y) := rank M + ind V, fi*(Y, Y) : = rank M + ind(-V), (1.39) where M := {I- X*X)W(Y,Y), V := V[W(Y,Y)]TX*XV, V := I - M] M (1.40) are n x n matrices with W(Y,Y) := YTJY being the Wronskian of Y and Y. The 2n x 2n matrix J is defined in (1.1). Here 'vaAV denotes the index (i.e., the number of negative eigenvalues) of the symmetric matrix V. Originally, the comparative index was developed for the discrete oscillation theory (see below) due to its connection with the "discrete focal points" [27,32,33]. This new approach allowed to solve several difficult open problems pertaining e.g. exact Sturmian separation and comparison theorems on compact interval (Section 2.2), a detailed distribution of focal points of conjoined bases throughout the given interval (Theorem 2.8), singular Sturmian separation and comparison theorems on unbounded intervals (Section 2.3), or the development of the oscillation theory of system (H) without the Legendre condition (1.3) in [38-40]. Comparative index also produced several important generalizations in the spectral theory [37-39]. The definition of the comparative index shows that it can be expressed in terms of the Riccati quotients as in (1.27), which are associated with the matrices Y and Y. More precisely, let Q and Q be any symmetric n X n matrices such that XTQX = XTU, XTQX = XTU. (1.41) Section 1.5. Comparative index 13 For example, according to (1.26) we can choose the symmetric matrices Q:=XX]UX], Q:=XX]UX]. (1.42) Then the matrix V in (1.40) has the form V = VXT(Q - Q)XV, (1.43) see e.g. [27, Theorem 3.2(iii)]. In particular, if the matrices X and X are invertible, then M. = 0, V = I, and equations (1.39), (1.42), and (1.43) yield that fi(Y,Y) = md(Q-Q), fi*(Y,Y)=md(Q-Q), Q:=UX~\ Q:=UX-\ (1.44) In the next chapter we will demonstrate the utility of the comparative index and the dual comparative index for the development of a precise Sturmian theory of system (H). CHAPTER 2 Sturmian theory for linear Hamiltonian systems Oscillation theory of linear Hamiltonian systems and Sturm-Liouville differential equations represents a classical topic in the qualitative theory of differential equations. Standard references include the monographs [11,21,30,46,58,67,69] by Atkinson, Coppel, Elias, Hartman, Kratz, and Reid, or more recently [9,56,70] by Amrein et al., Johnson, Obaya, Novo, Nunez, Fabbri, and Rofe-Beketov and Kholkin. In this chapter we present an overview of the Sturmian separation and comparison theorems for linear Hamiltonian systems and our recent contributions to this subject. Linear Hamiltonian systems (H) without the complete controllability assumption are intensively studied in the literature. For example, Johnson, Novo, Nunez, and Obaya proved in [54, Theorem 3.6] a formula connecting the rotation number of system (H) with the number of left proper focal points of a conjoined basis (X,U) of (H) in (a, b] when b —> oo. Uncontrollable systems (H) were also considered in [41,53,55] in the relation with the notion of a weak disconjugacy of (H) and dissipative control processes, and in [66,76,79-83] when studying the principal solutions of (H) at infinity. Let us mention that the transformation theory of linear Hamiltonian systems developed by Došlý and Elyseeva in [23-25,35, 36] is an important tool in the investigations of their qualitative properties, both in the context of the controllable and uncontrollable systems. In this chapter we consider nonoscillatory linear Hamiltonian systems, as defined in [92]. If the systems are oscillatory, then it is possible to measure a comparison of two such systems by means of the concept of relative oscillation. For completely controllable systems it was developed by Došlý in [26], which was extended to possibly uncontrollable systems by Elyseeva in [38]. 2.1. Review of Sturmian theory for controllable systems Regarding the compact interval X = [a, b], basic Sturmian separation and comparison theorems for the second order Sturm-Liouville differential equations are presented in [46, Theorem XI.3.1] or [69, Theorem 11.3.2(a)]. An extension of these results to completely controllable linear Hamiltonian systems (H) was derived in [20, Theorem 4] by Coppel, in [10, pg. 252] by Arnold (also quoted in [70, Theorem 4.8]), and in [58, Section 7.3] by Kratz. The notion of a completely controllable system (H) was defined in Section 1.1. In particular, we remark that Kratz proved in [58, Theorem 4.1.3] the following result. Proposition 2.1. Assume that the Legendre condition (1.3) holds on the interval X. Then system (H) is completely controllable on X if and only if for every conjoined basis (X,U) of (H) the matrix X(t) is singular only at isolated points in the interval X. As it is demonstrated in the above proposition, the development of the classical Sturmian theory for linear Hamiltonian system (H) is based on two assumptions. Namely, • the validity of the Legendre condition (1.3) on the interval [a, b] and • the complete controllability (or the identical normality) of system (H) on [a, b]. The above result justifies the following definition. A point íq £ X is a focal point of a conjoined basis (A, U) of (H) if X(to) is singular, and then m(t0) := def X(t0) = dimKerX(t0), m(t0) < n, (2.1) is its multiplicity. Here we use the terminology defect of a matrix (denoted by def) for the dimension of its kernel. For convenience we denote by mil) and mil) the total number of focal points of the 15 16 Chapter 2. Sturmian theory conjoined bases (A, U) and (X, U) of (H) in the interval X, that is, m(X) := ™(*), := S™(*), (2-2) where m(£o) is the multiplicity of the focal point to °f (A, ^) defined according to (2.1). Under the Legendre condition (1.3) the sums in (2.2) are finite when the interval X is compact or when the interval X is unbounded from above and system (H) is nonoscillatory. If we deal with an interval X with specific endpoints, such as the intervals X = (a, b] or [a, b) or (a, b) or (a, oo) etc., then we write m(a,b] or in (a, b] etc. for simplicity in the corresponding context. 2.1.1. Sturmian separation theorems. In [67, Corollary 1, pg. 366] or [69, Corollary 1, pg. 306], Reid proved the following Sturmian separation theorem for a completely controllable system (H) on an arbitrary bounded interval X. The numbers of focal points in the interval X of any two conjoined bases (X, U) and (X, U) of system (H) differ by at most n (which is the maximal multiplicity of a focal point), i.e., |m(X) - m(X)| < n. (2.3) Moreover, an improved estimate was derived in [69, Corollary 3, pp. 307-308] saying that |m(X) — m(X)| < n — m, (2-4) where m is the defect (i.e., the dimension of the kernel) of the Wronskian of the two conjoined bases (X,U) and (X,U). In addition, for one conjoined basis (X,U) of (H) the difference between the numbers of its focal points in (a, b] and in [a, b) equals to the value def X(b) - def A(a) = rank A(a) - rank A(6). (2.5) Specific results were also obtained for the principal solution (Xs, Us) of system (H) at the point s £ X, which we defined in (1.12). In this case we denote the number of focal points of (Xs, Us) in the interval X by ms(I) in the spirit of (2.2). Then the result in [67, Corollary 2, pg. 366] or [69, Corollary 2, pg. 307] states that for any conjoined basis (X, U) of system (H) we have ma(a, b) — n < m(a, b) < ma(a, b) + n, (2-6) ma(a, b] — n < m(a, b] < ma(a, b] + n, (2-7) while from [69, Theorem 8.3] we obtain that ma(a,b) = mh(a,b), ma(a,b] = mh[a,b). (2.8) As a continuation of the above results, Kratz derived in [58, Section 7.3] exact formulas for the difference of focal points in the open interval X = (a, b) of two conjoined bases (A, U) and (A, U) of system (H) by using the index (i.e., the number of negative eigenvalues) of the difference between the associated Riccati quotients. Namely, the results in [58, Theorem 7.3.1, pg. 194] states that m(a, b) - ih(a, b) = ind(Q - Q)(b~) - ind(Q - Q)(a+), (2.9) m(a, b) - ih(a, b) = ind(Q - Q)(a+) - ind(Q - Q)(b~), (2.10) where Q(t) := U(t) A_1(t) and Q(t) := U(t) A"1^) according to (1.25) and where ind(Q - Q)(t^) denote the one-sided limits of the index of the matrix Q(t) — Q(t). Note that ind[Q(t) — Q(t)] is a piecewise constant quantity in the interval (a, b) under the Legendre condition (1.3) and that the difference on the right-hand side of (2.9) and (2.10) is always less or equal to n, which complies with the earlier estimate by Reid in (2.3). Moreover, by using the principal solutions at a and b, Kratz obtained in [58, Corollary 7.3.2, pg. 196] that for any conjoined basis (X,U) of (H) we have m(a, b) = ma(a, b) + ind(Qa - Q)(b~) > ma(a, b), (2-11) m(a, b) = mfJ(a, b) + ind(Q — Qb)(a+) > mb(a, b), (2.12) where Qs(t) := Us(t) A"1^). Section 2.1. Sturmian theory for controllable systems 17 A singular Sturmian separation theorem for a completely controllable and nonoscillatory system (H) on X = [a, oo) was derived in ['. i] by Došlý and Kratz. The main ingredient is the concept of the principal solution of system (H) at infinity, which we discussed in Section 1.4. Following (1.31), the principal solution of (H) at infinity is defined as a conjoined basis (Aoo, ř/oo) such that Aoc(i) is invertible on an interval [a, oo) for some a > a and the matrix ^ = 0, where := hm (^j X^{s) B{s) X^(s) ds^j . (2.13) The principal solution (A^, řTco) at infinity exists and is unique (up to a constant right invertible multiple) by Proposition 1.19. Then in [29, Theorem 1] it is shown that, under the Legendre condition (1.3), for any conjoined basis (A, U) of (H) we have the estimate m[a, oo) > moo [a, °°)? (2-14) where m^a, oo) denotes the number of focal points of the (unique) principal solution of (H) at infinity in the indicated interval [a,oo). Moreover, the result in [29, Corollary 1] states that if the interval X = R = (—00,00), then the numbers of focal points of the principal solutions of (H) at infinity and at minus infinity in the whole interval R satisfy m00(-00, 00) = m_oo(—00, 00), (2.15) which is a singular version of the first equality in (2.8). Moreover, by taking the limit for a —> —00 in (2.14) we obtain for any conjoined basis (A, U) of (H) the estimate m(—00, 00) > moo(—00, 00). (2.16) 2.1.2. Sturmian comparison theorems. Next we review the Sturmian comparison theorems, which provide estimates for the numbers of focal points of two conjoined bases of two possibly different completely controllable linear Hamiltonian systems. Thus, together with system (H) we consider another linear Hamiltonian system ý' = jH(t)ý, tel, (H) where the coefficient matrix ~H : X —> K2rax2ra is piecewise continuous and symmetric on X. Moreover, we assume that the matrices ~H(t) in system (H) and ~H{t) in system (H) are related by the Sturmian majorant condition U{t) > U{t) for all t e X. (2.17) In this setting we say that system (H) is a Sturmian majorant of (H), or that system (H) is a Sturmian minorant of (H). In addition to (2.17) we assume that the minorant system (H) satisfies the Legendre condition B(t)>0 foralHeX. (2.18) Here B(t) is the lower right nxn block of "Hit), i.e., we partition the matrix ~H{t) similarly to (1.1) as «M=f-£(? tel. (2.19) A(t) B(t) where A(t), B(t), C(t) are piecewise continuous nxn matrix-valued functions on X with B(t) and C(t) being symmetric for t G 1. Assumption (2.17) then implies that the Legendre condition (1.3) holds also for the majorant system (H). Given a conjoined basis (A, U) of system (H), we denote by m(X) the total number of its focal points in the interval X, following the notation in (2.2). In [20, Theorem 4], Coppel derived under assumptions (2.17) and (2.18) a comparison result on a bounded interval X for the principal solutions (Xa, Ua) and (Xa, Ua) of systems (H) and (H) at the point a in the form ma(a,b] > ma(a,b}. (2.20) Such estimates are also known in the works by Arnold in [10, pg. 252], which is also quoted in [70, Theorem 4.8] by Roffe-Beketov and Kholkin. However, the latter two references use the majorant condition (2.17) together with the strengthened Legendre condition B(t) > 0 on the interval X. On the other hand, in [58, Theorem 7.3.1, pg. 194], Kratz derived the Sturmian comparison theorem for 18 Chapter 2. Sturmian theory the open interval X = (a, b) saying that for any two conjoined bases (A, U) and (X, Ú) of systems (H) and (H) we have the estimates m(a, b) - m(a, b) > ind(Q - Q)(6~) - ind(Q - Q)(a+), (2.21) m(a, b) - m(a, b) > ind(Q - Q)(a+) - ind(Q - Q)(b~), (2.22) where Q(t) := ř7(i) A_1(i) and Q(t) := U(t) A_1(i) as we discussed above. Moreover, by using the principal solutions at a and b of system (H) in the place of (X,U), Kratz obtained in [58, Corollary 7.3.2, pg. 196] that for any conjoined basis (X,U) of (H) we have the estimates m(a, b) > rha(a, b) + ind(<5a - Q)Q>~) > fha(a, b), (2.23) m(a, b) > rhb{a, b) + md(Q - Qb)(a+) > mh(a, b), (2.24) where Qs(t) := Us(t) X~1(i). Note that according to the Sturmian separation theorem in (2.8) applied to system (H) we have fhb(a, b) = rha{a, b) in (2.24). By taking (X, U) := (Xa, Ua) being the principal solution of system (H) at a we obtain from (2.23) and (2.24) the inequality ma(a, b) > ma(a, b), (2.25) which complements the earlier estimate in (2.20) by Coppel. Regarding an open or unbounded interval X, a singular Sturmian comparison theorem for the second order Sturm-Liouville differential equations was obtained in [1, Theorem l(i)] by Aharonov and Elias. Moreover, a singular comparison theorem for completely controllable and nonoscillatory systems (H) and (H) onX = [a, oo) was derived in [29, Theorem 2] by Došlý and Kratz. More precisely, for any conjoined basis (A, U) of (H) we have the estimate m[a, oo) > mcoja, oo), (2.26) where moo [a, °°) denotes the number of focal points of the (unique) principal solution of the minorant system (H) at infinity. Moreover, if assumptions (2.17) and (2.18) hold on the unbounded interval X = (—00,00), then we obtain from (2.26) by taking the limit for a —> —00 for any conjoined basis (A, U) of (H) the estimate m(—00,00) > m00(—00, 00). (2.27) Note that additional results about the singular Sturmian comparison theorems for completely controllable systems (H) and (H) will be presented in Corollary 2.26 and in Subsection 2.3.3. In the next sections we will show how the above estimates are generalized to possibly abnormal systems (H) and (H) on bounded and/or unbounded intervals X. Moreover, we will also see that several results regarding possibly uncontrollable linear Hamiltonian systems are new even for systems, which are completely controllable. 2.2. General Sturmian theory on compact interval The Sturmian separation and comparison theorems on the compact interval X for possibly abnormal (or uncontrollable) systems (H) were derived in [60, Corollary 4.8] and [91, Theorems 1.2-1.5] by Kratz and Simon Hilscher. This new theory employs the notion of a generalized (or proper) focal point of a conjoined basis of (H), which was introduced in [59] by Kratz and subsequently more specified [96] by Wahrheit, who defined the corresponding multiplicities of left and right proper focal points, see equations (2.30) and (2.31) below. In this section we consider the compact interval X = [a, b]. In the previous section we saw that the classical Sturmian theory for linear Hamiltonian system (H) is based on the complete controllability assumption. When this assumption is removed, Kratz and independently Fabbri, Johnson, and Núňez showed in [59, Theorem 3] and [42, Proof of Lemma 3.6(a)] the following result. Proposition 2.2. Assume that the Legendre condition (1.3) holds on X = [a, b]. Then for any conjoined basis (A, U) of system(R) the kernel of X(t) is piecewise constant on [a, b]. More precisely, for Section 2.2. Sturmian theory on compact interval 19 a given conjoined basis (X, U) of (H) there exists a partition a = to < t± < ■ ■ ■ < tm = b such that KerX(t) is constant on the open interval (tj,tj+i) for all j £ {0,1,... , m — 1} and KeiX(tj) C KeiX(tj), j £ {1, 2,..., m}, (2.28) KerX(t+) C KeiX(tj), j £ {0,1,...,m - 1}. (2.29) The quantities KerX(t^) in (2.28) and (2.29) denote the limits of the constant set KerX(t) as t —> t^. The inclusions in (2.28) and (2.29) follow from the continuity of the matrix X(t) on [a, b]. In the subsequent work [96], Wahrheit defined the point to £ (a,b] to be a left proper focal point of (X, U) if Ker X(t^) C Ker X(t0), with the multiplicity mL(t0) := def X(t0) - def ) = rankA^) - rankX(t0). (2.30) In a similar way we define to £ [a? to be a rzg/it proper focal point of (X, U) by the condition Ker X(t^) ^ Ker X(to), with the multiplicity mR(t0) := deiX(t0) - defX(t£) = rankX(i^) - rankX(t0). (2.31) The notations def X(t^) and rankX(tQt) represent the one-sided limits at to of the piecewise constant quantities defX(t) and rankX(t). Let (X,U) and (X,U) be two conjoined bases of system (H). Moreover, given a point s £ [a, b] let (Xs, Us) be the principal solution of (H) at the point s £ [a, b] as we defined in (1.12). We set y(*>:=($o)' *® ■= (IS)' y^:=(SS)' feM]' (2-32) which are 2n x n matrix solutions of system (H). This notation will be used in particular when we deal with conjoined bases of system (H) combined together with the comparative index. It will be also useful for calculating the Wronskian of two conjoined bases, as it is shown in (1.38). For convenience we denote by mi,(a,b], fhi,(a,b], and mi,s(a,b] the total number of left proper focal points of Y, Y, and Ys in the half-open interval (a, b], respectively. Similarly, we denote by mji[a, b), m/j[a, b), and mjjs[a, b) the total number of right proper focal points of Y, Y, and 1^ in the half-open interval [a, b), respectively. We note that the left and right proper focal points are always counted including their multiplicities. By (2.30) and (2.31) we then have the equalities mL(a,b]= ^mL(t), mR[a,b) = ^mR(t). (2.33) tG(a,6] tG[a,6) Under (1.3) these sums are always finite, compare with the notation in (2.2). The first Sturmian separation theorems for a possibly uncontrollable system (H) on bounded interval X were derived in [91] by Simon Hilscher by using the eigenvalue theory for a certain perturbed linear Hamiltonian system. In [91, Theorem 1.4] and [60, Remark 4.7] it is shown that under (1.3) for any conjoined basis (X, U) of system (H) we have the estimates mLa(a, b] < mL(a, b] < mLa(a, b] + n, (2.34) mRb[a, b) < mR[a, b) < mRh[a, b) + n. (2.35) These inequalities imply, see [91, Theorem 1.5], that for any conjoined bases (X,U) and (X,U) of (H) we have the estimates |m£,(a, b] — m£,(a, b] | < n, (2.36) \mR[a,b) - fhR[a, b)\ < n, (2.37) which generalize the results in (2.3) and (2.7) to possibly uncontrollable systems. Moreover, from [60, Corollary 4.8] we know that mLa(a, b] = mRh[a, b). (2.38) The first Sturmian comparison theorems for two possibly uncontrollable systems (H) and (H) satisfying the majorant condition (2.17) and the Legendre condition (2.18) were also derived in [91]. Note that 20 Chapter 2. Sturmian theory the roles of the systems (H) and (H) in [91] are interchanged. More precisely and with the notation in (2.17), the result in [91, Theorem 1.2] states that for any conjoined basis (X, U) of system (H) we have the estimate mL(a, b] < mLa(a, b] + n, (2.39) while [91, Theorem 1.3] states that for any conjoined basis (X, U) of (H) we have the estimate mL(a,b] > mLa(a,b]. (2.40) By using [60, Remark 4.7] the estimates in (2.39) and (2.40) can be reformulated as mR[a, b) < mRh[a, b) + n, (2-41) mR[a,b) > mRb[a,b). (2.42) We will comment about our contribution to this subject in Section 2.3. 2.2.1. Sturmian separation theorems. Using the above notation we can formulate a precise Sturmian separation theorem for system (H), which involves the comparative index when dealing with the left proper focal points and the dual comparative index when dealing with the right proper focal points. In this subsection we present the mains results from [84, Sections 4-6], see Appendix C. We also add some closely related results from [89, Section 1 and 3]. The next result was independently obtained also in [34, Theorem 2.3] by Elyseeva. Theorem 2.3 (Sturmian separation theorem). Assume that (1.3) holds. Then for any conjoined bases Y and Y of (H) we have the equalities mL(a, b] - fhL{a, b] = »(Y(b),Y(b)) - »(Y(a), Y(a)), (2.43) mR[a, b) - mR[a, b) = M* (Y(a), Y(a)) - M* (Y(b), Y(b)). (2.44) In the following we provide a formula, which relates the number of left proper focal points in (a, b] and the number of right proper focal points in [a, b) for one conjoined basis of (H). This extends the information provided in (2.5) to possibly uncontrollable system (H). Theorem 2.4. Assume that (1.3) holds. Then for any conjoined basis (X,U) of (H) its numbers of left proper focal points in (a, b] and right proper focal points in [a, b) satisfy rri£,(a, b] + rank X(b) = mR[a, b) + rank X(a). (2.45) The next two results demonstrate that the principal solutions Ya and Yb of system (H) at the points a and b play a prominent roles in the presented Sturmian theory of (H). The numbers mLa(a,b], mRa[a,b), mLb(a,b], mRb[a,b) (2.46) turn out to be essential parameters of system (H) on the interval [a, b]. More precisely, they are optimal bounds for the numbers of left and right proper focal points of any conjoined basis (X, U) in (a, b] and [a, b), respectively. In the next results we use the notation from (2.32) regarding the principal solutions Ya and Yf, of (H). Theorem 2.5. Assume that (1.3) holds. With the notation in (2.46) we have for the left and right focal points of the principal solutions (Xa,Ua) and (Xb,Ub) in (a,b] and [a, b), respectively, the equalities mLh(a,b] =mLa(a,b] +rankXa(6) = mLa(a,b] +rankX6(a), (2.47) mRa[a,b) = mRb[a,b) +rankX6(a) = mRb[a,b) +rankXa(6), (2.48) mRa[a,b) = mLb(a, b], mLa(a,b] = mRb[a,b). (2.49) Note that the second equality in (2.49) is known in (2.38), while the first equality in (2.49) is new. Theorem 2.6 (Sturmian separation theorem). Assume that (1.3) holds. Then for any conjoined basis (X, U) of (H) we have the inequalities mLa(a, b] < mL(a, b] < mLb(a, b], (2.50) mRb[a, b) < mR[a, b) < mRa[a, b). (2.51) Section 2.2. Sturmian theory on compact interval 21 According to (2.49) in Theorem 2.5, the lower bounds in (2.50) and (2.51) are the same, as well as the upper bounds in (2.50) and (2.51) are the same. Moreover, these lower and upper bounds are independent on the conjoined basis (A, U). Since these bounds are attained for the specific choices of (A, U) := (Aa, Ua) and (A, U) := (Xf,, Ub), the inequalities in (2.50) and (2.51) cannot be improved -in the sense that the estimates (2.50) and (2.51) are satisfied for all conjoined bases (A, U) of (H). In the context of Theorem 2.3 the principal solutions Ya and Yf, can be viewed as reference solutions of system (H) when counting the number of left and right proper focal points in (a, b] and [a, b), respectively. More precisely, every conjoined basis Y of system (H) satisfies the exact formulas mL(a, b] = mLa(a, b] + fi(Y(b), Ya(b)), (2.52) mR[a, b) = mRb[a, b) + fi*(Y(a), Yb(a)). (2.53) Inequalities (2.50)-(2.51) together with (2.47)-(2.48) imply directly the optimal and the universal bound for the difference between the numbers of proper focal points of any two conjoined bases of system (H). This result improves the estimates in (2.36) and (2.37). Note that this result is also new in the controllable case, where it generalizes the estimate presented in (2.3). Corollary 2.7 (Sturmian separation theorem). Assume that (1.3) holds. Then for any conjoined bases (X,U) and (X,U) of (H) we have the estimates | m£,(a, b] — m£,(a, b] | < rankXa(b) = rank Xb{a) < n, (2.54) | mR[a,b) — rhR[a,b) | < rankA^a) = rankAa(6) < n, (2.55) | m£,(a, b] — fhR[a,b) | < rankAa(6) = rank X;,(a) < n. (2.56) The results in Theorem 2.6 also pose the natural question, whether for any given integers £ and r within the lower and upper bounds in (2.50) and (2.51) there exists a conjoined basis Y of system (H), for which the equalities mi,(a,b] = £ and mR[a,b)=r (2.57) hold. And if so, then how to determine such a conjoined basis. The answers to both these questions are presented in the next statement from [89, Theorem 1.1]. Theorem 2.8. Assume that (1.3) holds. Then for any integers £ and r satisfying mLa(a,b] < £ < mLh(a,b] and mRh[a,b) < r < mRa[a,b) (2.58) there exists a conjoined basis Y of (H) such that (2.57) holds. Moreover, if £>r, then the conjoined basis Y can be chosen with X(a) = I, and if £ < r, then the conjoined basis Y can be chosen with X(b) = I. In particular, when £ = r the conjoined basis Y may be chosen with both X(a) and X(b) invertible. In the case when system (H) is completely controllable on [a, b], then Theorem 2.8 represents a generalization of the classical result by Reid, see e.g. [69, Theorem V.6.3, pg. 284-285] or [67, Theorem VII.5.1] in combination with [59, Theorem 1]. We display this result explicitly for an easy comparison with the results in Section 1.3 regarding the explicit Riccati equation (R), see [89, Theorem 3.2 and Remark 3.3] for more details. Theorem 2.9. Assume that (1.3) holds and system (H) is completely controllable on [a,b]. Then the following statements are equivalent. (i) There exists a conjoined basis (X,U) of (H) such that X(t) is invertible on (a, b]. (ii) For any integer r with 0 < r < n there exists a conjoined basis (A, U) of (H) such that X(t) is invertible on (a, 6] and m(a) = r. (iii) There exists a conjoined basis (X,U) of (H) such that X(t) is invertible on [a, b). (iv) For any integer £ with 0 < £ < n there exists a conjoined basis (A, U) of (H) such that X(t) is invertible on [a,b) and m(b) = £. 22 Chapter 2. Sturmian theory The results in Theorem 2.9 are important for applications. For example, in the theory of Riccati matrix differential equations, see e.g. [58,67,68,77], they provide a sufficient and also a necessary condition for the existence of a symmetric solution of (R) on the whole intervals [a, b], (a,b], or [a, b) by considering the Riccati quotient Q(t) = U(t) A_1(i) in (1.25). We remark that the conjoined basis (X, U) in part (ii) of Theorem 2.9 can be constructed by prescribing the initial conditions at the point b. More precisely, all such conjoined bases (X, U) (up to a constant right nonsingular multiple) are determined as X(b) = I, U{b) = D + Qa(b), D < 0, indD = rankD = n — r, (2.59) where Qa is the Riccati quotient in (1.26) associated with the principal solution Ya. Similarly, all conjoined bases Y (up to a constant right nonsingular multiple) in part (iv) of Theorem 2.9 are constructed by the initial conditions at the point a. Namely, we have X(a) = I, U{a) = D+ Qb(a), D > 0, ind(-D) = rankD = n - £, (2.60) where Qf, is the Riccati quotient in (1.26) associated with the principal solution Yb. The above results from the Sturmian theory of system (H) allow to derive additional properties of the comparative index involving two conjoined bases of (H). These properties are of a local character and they are expressed in terms of the limit of the comparative index, see [84, Theorems 6.1 and 6.3] and [34, Theorem 2.3]. Theorem 2.10. Assume that (1.3) holds. Then for any two conjoined bases Y and Y of system (H) the following properties are satisfied. (i) The comparative index fi(Y(t),Y(t)) is piecewise constant on [a,b] and right continuous on [a, b). In addition, lim fi(Y(t),Y(t)) =fi(Y(t0),Y(t0)) -mL{t0)+mL{t0), t0 e (a,b], (2.61) and fj,(Y(t),Y(t)) is not left continuous at to G (a, b] if and only ifmi,(to) ^ friLito). (ii) The dual comparative index /i* (Y(t), Y(t)) is piecewise constant on [a,b] and left continuous on (a, b]. In addition, lim »*(Y(t),Y(t)) =»*(Y(to),Y(to)) -mR(to)+mR(to), t0 G [a, b), (2.62) and fj,*(Y(t),Y(t)) is not right continuous at to G [a, b) if and only if mR{to) 7^ fhR{to). The next result shows how to compute the multiplicities in (2.30) and (2.31) of left and right proper focal points of Y at some point to by a limit involving the comparative index. Theorem 2.11. Assume that (1.3) holds. Let Y be a conjoined basis of (H) and let Yt be the principal solution of (H) at the point t in (2.32). Then we have mL{t0) = \im_n(Y(to),Yt(t0)), t0 G (a,6], (2.63) mR{t0) = lim [A* (Y(to),Yt (to)), t0 G [a, b). (2.64) 2.2.2. Sturmian comparison theorems. In this is subsection we comment on the Sturmian separation theorems (in particular on the results in Theorems 2.3 and 2.10) from a more general context. More precisely, we present the Sturmian comparison theorem for conjoined bases of two systems of the form (H) and (H) satisfying the majorant condition (2.17). Along with the basic systems (H) and (H) we will also consider a certain transformed linear Hamiltonian system y' = jn(t)y, tel, (H) which is related to (H) and (H) by a symplectic transformation, see formula (2.70) below. The following exact formula for expressing the numbers of left and right proper focal points of conjoined bases Y and Y of systems (H) and (H) on X = [a, b] was derived in [34, Theorem 2.2] by Section 2.2. Sturmian theory on compact interval 23 Elyseeva. In the spirit of (2.33), we denote by fh^ia, b] and mjj[a, b) the total number of left and right proper focal points of the conjoined basis Y of system (H) in the indicated interval. Similar notation friL^a, b] and fhR[a,b) will be used for the conjoined basis Y of system (H), see below. Under (1.3) these sums are always finite. For convenience we define the constant 2n x n matrix E := (0, If, (2.65) which can be considered, in view of (1.12), as the initial condition for the principal solutions Ys, Ys, Ys of systems (H), (H), (H) at the point s£l Theorem 2.12. Assume that (2.17) and (2.18) hold onI= [a,b] and let Y and Y be any conjoined bases of (H) and (H). Let Z be a fundamental matrix of (H) satisfying Y(t) = Z(t) E on [a, b], where the matrix E is given in (2.65), and consider the function Y(t) := Z~1(t)Y(t) on [a,b\. Then the comparative index fi(Y(t),Y(t)) is piecewise constant on [a,b] and right-continuous on [a,b) and for every to £ (a, b] the multiplicities m^ito), rh^ito), and friL,(to) of left proper focal points ofY, Y, and Y at to defined through (2.30) satisfy the equality mL{t0) - mL{t0) = mL{t0) + fi(Y(t0),Y(t0)) - lim fi(Y(t),Y(t)). (2.66) Moreover, the numbers of left proper focal points of Y and Y in (a, b] are connected by mL(a, b] - mL(a, b] = mL(a, b] + »(Y(b), Y(b)) - »(Y(a),Y(a)), (2.67) where rh^ia, b] is the number of left proper focal points in (a, b] of the auxiliary function Y. A corresponding result for the right proper focal points in [a, b) can be derived by an analogous method to the proof of Theorem 2.12 in [34]. Alternatively, we may use the relationship in (2.45) in Theorem 2.4 between the left and right proper focal points of Y. Theorem 2.13. Under the assumptions of Theorem 2.12, for any conjoined bases Y and Y of (H) and (H) the dual comparative index fi*(Y'(t),Y'(t)) is piecewise constant on [a,b] and left-continuous on (a,b] and for every to £ [a,b) the multiplicities mRito), fhRito), and mRito) of right proper focal points ofY, Y, and Y := Z_1Y at to defined through (2.31) satisfy the equality mR(to) - mR(to) = mR(to) + fi*(Y(t0), Y(t0)) - lim »*(Y(t),Y(t)). (2.68) Moreover, the numbers of right proper focal points of Y and Y in [a, b) are connected by mR[a, b) - mR[a, b) = rhR[a, b) + »*(Y(a), Y(a)) - »*(Y(b), Y(b)), (2.69) where mjj[a, b) is the number of right proper focal points in [a, b) of the auxiliary function Y. We note that the symplectic fundamental matrix Z of (H) in Theorems 2.12 and 2.13 has the form Z = (* , Y). Moreover, it is easy to verify (see [26]) that the function Y := Z~XY is a conjoined basis of the transformed linear Hamiltonian system (H), whose coefficient matrix U{t) := ZT(t)[n(t)-n(t)]Z(t), tel, (2.70) satisfies ~H(t) > 0 on X under (2.17). In particular, the Legendre condition B(t) > 0 for all i £ X (2.71) holds, where B(t) is the lower right n x n block of the matrix ~H(t). Condition (2.71) implies that the quantities m£,(io), "ifj(io) m (2.66), (2.68) and the quantities rh^ia, b], mjj[a,6) in (2.67), (2.69) are correctly defined. When the two systems (H) and (H) coincide, i.e., when ~H(t) = ~H(t) on X, then ~H(t) = 0 on X by (2.70) and hence, all conjoined bases Y of (H) are constant on X and we have m£,(a, b] = 0 = fhR[a, b). In this case the results in (2.67) and (2.69) reduce to formulas (2.43) and (2.44) in Theorem 2.3. Similarly, the results in (2.66) and (2.68) reduce to formulas (2.61) and (2.62) in Theorem 2.10, since in this case we have fhi,(to) = 0 = mfj(io)- 24 Chapter 2. Sturmian theory 2.3. General Singular Sturmian theory on unbounded intervals In this section we present our fundamental contributions to the singular Sturmian theory for nonoscillatory and possibly uncontrollable linear Hamiltonian systems (H) on the unbounded interval X = [a, oo). These new results employ two key tools, namely, • the theory of minimal principal and maximal antiprincipal solutions of (H) at infinity, and • the concept of a multiplicity of a focal point at infinity. In [79,81,85] we showed that every conjoined basis Y of (H) with constant kernel on [a,oo) with d[a, oo) = dm satisfies n — dec < rankX(t) < n, i£[a,oo), (2.72) 0 < rankTQ>00 < n - d^, TQ>00 := lim t—>oo JtX^(s)B(s)X^T(s)ds^. (2.73) We recall from Section 1.4 that Y is the minimal principal solution at infinity if the corresponding matrix in (2.73) satisfies TQOO = 0 and if rankX(t) = n — d^ on [a,oo). Moreover, according to [81, Definition 5.1], a conjoined basis Y of (H) with constant kernel on [cy, oo) with c?[cy, oo) — ^oo is a maximal antiprincipal solution at infinity if eventually rankX(t) = n and the corresponding matrix TQ>00 in (2.73) satisfies rankTQ>00 = n - d^, i.e., the rank of the matrix Ta ^ is maximal according to (2.73). In this section we will use the notation Yoo for the minimal principal solution at infinity (in Section 1.4 we used the notation (Xmin, ?7min)), while for a maximal antiprincipal solution at infinity we will use the notation Y^. By [87, Proposition 2.3] we know that a given maximal antiprincipal solution Y^ completes the minimal principal solution Yqo (or its suitable invertible multiple) to a symplectic fundamental matrix of system (H). That is, we have Zoo{t) = (Yooit) l^o(t)), t£[a,oo), W(Y00,Y00) = I, (2.74) where WiY^, Yx>) is the Wronskian of the conjoined bases Y^ and Yx>- Then every conjoined basis Y of (H) can be uniquely represented by a constant 2n X n matrix C^, satisfying Y(t) = Z00(t)C00, t£[a,oo), Coo:=^^2y))- ^ The following notion appeared in [86] and it is completely new in the theory of linear Hamiltonian differential systems. It provides a unified view on the principal solutions of system (H) at a finite point and at infinity, see equality (2.77) in Theorem 2.15 below. Definition 2.14 (Multiplicity of focal point at infinity). Let Y be a conjoined basis of system (H) with constant kernel on the interval [a, oo) for some a £ [a^, oo) with defined in (1.7). We say that Y has a (left) proper focal point at infinity if + rankTQOO < n with the multiplicity mi(oo) := n — d^ — rankTQj00, (2.76) where d^ is the maximal order of abnornality of (H) in (1.6) and TQj00 is the matrix defined in (2.73) corresponding to Y. In accordance with (2.73) we note that under (1.3) the number 112^(00) defined in (2.76) is always nonnegative. Moreover, it does not depend on the particular choice of the point a £ [000,00), for which the conjoined basis Y has constant kernel on [a, 00). In particular, we have the estimates 0 < mi(oo) < n — doc. It follows that the conjoined basis Y has no focal point at infinity, i.e. mi(oo) = 0, if and only if Y is an antiprincipal solution of (H) at infinity. Similarly, the multiplicity mi(oo) = n — dx is maximal possible if and only if Y is a principal solution of (H) at infinity. The next result, see [86, Theorem 3.3], provides a way for computing the multiplicity of the focal point at infinity in terms of the rank of the genus of a conjoined basis Y and the rank of the Wronskian of Y with the minimal principal solution Yx at infinity. Section 2.3. Singular Sturmian theory 25 Theorem 2.15. Assume that (1.3) holds with [a, oo) and system (H) is nonosdilatory. Let Y be a conjoined basis of (H) belonging to a genus Q. Then the multiplicity of the focal point of Y at infinity defined in (2.76) satisfies the formula mL(oo) = rankQ - rank W(Yoo, Y), (2.77) where the quantity rankC/ is defined in (1.10) and W(Yoc,Y) is the Wronskian ofXx, and Y. Remark 2.16. When system (H) corresponds to the second order Sturm-Liouville differential equation, then the statement of Theorem 2.15 characterizes the principal solutions at infinity as those solutions with m£,(oo) = 1. On the other hand, all nonprincipal solutions at infinity satisfy mi(oo) = 0. 2.3.1. Singular Sturmian separation theorems. In this subsection we present the main results from [86, Sections 5-7], see Appendix D. More precisely, we provide Sturmian separation theorems for conjoined bases of a nonoscillatory system (H) on the unbounded intervals (a, 00], resp. [a, 00). We emphasize that the results regarding left proper focal points include the multiplicity of proper focal point at infinity, which was introduced in Definition 2.14. In addition, we will also provide the corresponding results for the open interval (a, 00). We note that the results in this subsection hold also with the left endpoint a = — 00 for the intervals X = (—00, b] or X = (—00,00), if we consider the corresponding concept of the minimal principal solution of (H) at minus infinity. A detailed analysis of this case is presented in [86, Remarks 5.16 and 8.1], see Appendix D. The following result corresponds to formulas (2.43) and (2.44), where a compact interval X = [a, b] was considered. It also justifies the necessity of including the multiplicity of proper focal point at infinity in the singular Sturmian theory of system (H). Theorem 2.17 (Singular Sturmian separation theorem). Assume that (1.3) holds with X = [a, 00) and system (H) is nonoscillatory. Then for any conjoined bases Y andY of (H) we have the equalities mL(a, 00] - fhL(a, 00] = /x(Coo, C^) - v(Y(a), Y(a)), (2.78) mR[a, 00) - fhR[a, 00) = ^* (Y(a), Y(a)) - (C^, Coo), (2.79) where Coo and Coo are the constant matrices in (2.75) corresponding to Y and Y. By considering a special choice of the conjoined basis Y in Theorem 2.17 we obtain formulas for the exact numbers of left and right proper focal points of any conjoined basis Y of (H) in the intervals (a, 00] and [a, 00). They highlight the importance of the minimal principal solution Yx> of (H) at infinity and the principal solution Ya of (H) at a in counting the numbers mi,(a, 00] and mR[a, 00). They correspond to formulas (2.52) and (2.53), where a compact interval X = [a, b] was considered. For convenience we denote by m,i00(a, 00] the total number of left proper focal points of the minimal principal solution Yx> of (H) at infinity in the interval (a, 00]. Similarly, we denote by mRoo[a,oo) the total number of right proper focal points of Yoo in the interval [a, 00). Theorem 2.18. Assume that (1.3) holds with I = [a, 00) and system (H) is nonoscillatory. Then for any conjoined basis Y of (H) we have the equalities mL(a, 00] = mLa(a, 00] + ^i{C00, C^), (2.80) mR[a, 00) = mRoo[a, 00) + (Y(a), l^o(a)), (2.81) where Coo and C^ are the constant matrices in (2.75) corresponding to Y and Ya. In the next statement we connect the multiplicities of left and right proper focal points of one conjoined basis Y of (H) in an unbounded interval. This result corresponds to formula (2.45). Theorem 2.19. Assume that (1.3) holds withl = [a, 00) and system (H) is nonoscillatory. LetYoo be the minimal principal solution of (H) at infinity. Then for any conjoined basis Y of (H) its numbers of left proper focal points in the interval (a, 00] and right proper focal points in the interval [a, 00) satisfy the equality mi(a, 00] + rank W(Yoo, Y) = mR[a, 00) + rank A(a). (2.82) 26 Chapter 2. Sturmian theory In the remaining results of this subsection we will use the principal solution Ya of (H) at the point a and the minimal principal solution of (H) at infinity as important ingredients in the presented results from the singular Sturmian theory on the unbounded interval X = [a, oo). The following statement relates the numbers of left and right proper focal points of Ya and Yx in (a, oo] and [a, oo). This result corresponds to formulas (2.47)-(2.49) in Theorem 2.5. Theorem 2.20. Assume that the Legendre condition (1.3) holds with X = [a, oo) and system (H) is nonosdilatory. Then we have the formulas mLQO(a, oo] = mLa(a, oo] + rankX00(a), mRa[a, oo) = mRoo[a, oo) + rankX^a), (2.83) mLa(a, oo] = mRoo[a,oo), mRa[a, oo) = mLoo(a, oo]. (2.84) We remark that equations (2.80) and (2.81) yield the lower bounds mL(a, oo] > mLa(a, oo], mR[a, oo) > mRoo[a, oo) (2.85) for the numbers of left and right proper focal points of any conjoined basis Y of (H) in the interval (a,oo] and [a, oo). These two lower bounds are the same according to (2.84). Moreover, the second estimate in (2.85) generalizes the result of Došlý and Kratz in (2.14) to possibly uncontrollable system (H). In the next statement we provide the corresponding optimal upper bounds for the numbers m£,(a, oo] and mR[a, oo). These estimates correspond to (2.50)-(2.51) in Theorem 2.6. Theorem 2.21 (Singular Sturmian separation theorem). Assume that (1.3) holds with X = [a, oo) and system (H) is nonos dilatory. Then for any conjoined basis Y of (H) we have mLa(a, oo] < mL(a, oo] < mLoo(a, oo], (2.86) mRoo[a, oo) < mR[a, oo) < mRa[a, oo). (2.87) The results in the above theorem yield the following optimal estimates for the difference of the numbers of the left and right proper focal points of any two conjoined bases of (H). They correspond to (2.54)-(2.56) in Corollary 2.7. Corollary 2.22 (Singular Sturmian separation theorem). Assume that (1.3) holds with X = [a, oo) and system (H) is nonos dilatory. Then for any conjoined bases Y and Y of (H) we have J m£,(a, oo] — m^ia, oo] | < rank X^ia) < n, J mR[a, oo) — rhR[a, oo) | < rank X00(a) < n, J m£,(a, oo] — mR[a, oo) | < rank X^ia) < n. In the next part of this subsection we analyze the numbers m£,(a, oo) and mR(a, oo) of left and right proper focal points of a conjoined basis Y of (H) in the open interval (a,oo). The motivation comes from possible practical applications, where the independent variable is always finite. In the following statement we provide optimal lower and upper bounds for the numbers left and right proper focal points of any conjoined basis Y of (H) in the open interval (a, oo). It is surprising that the optimal upper bounds for m^ia, oo) and mR(a, oo) are the same as in (2.86) and (2.87), i.e., they are equal to mij00ia1oo] and mRa[a, oo), respectively. Theorem 2.23. Assume that (1.3) holds with I = [a, oo) and system (H) is nonos dilatory. Then for any conjoined basis Y of (H) we have mLa(a, oo) < mL(a, oo) < mLoo(a, oo], (2.88) mRoo(a, oo) < mR(a, oo) < mRa[a, oo). (2.89) As an analogy of Corollary 2.22 we obtain from Theorem 2.23 the following estimates. Corollary 2.24. Assume that (1.3) holds with X = [a, oo) and system (H) is nonos dilatory. Then for any two conjoined bases Y and Y of (H) we have J m£,(a, oo) — m^ia, oo) | < rank Qa < n, (2.90) J mR(a, oo) — rhR(a, oo) | < rank Aoo(a+) < n, (2.91) Section 2.3. Singular Sturmian theory 27 where Qa is the genus of conjoined bases of (H), which contains the principal solution Ya. The next result connects the multiplicities of left and right proper focal points of the principal solutions Ya and Yx> in the open interval (a, oo). Theorem 2.25. Assume that (1.3) holds with X = [a, oo) and system (H) is nonosdilatory. Then we have the formula mLa(a, oo) + rank£a = mRQO(a, oo) + rankX00(a+), (2.92) where Qa is the genus of the principal solution Ya. In particular, the equality mLa(a, oo) = mRQO(a, oo) (2.93) holds if and only z/rankC/a = rankXco(a+). When system (H) is completely controllable then every conjoined basis Y of (H) has the matrix X(t) invertible near a. In addition, if (H) is nonoscillatory, then X(t) is also invertible near oo. Therefore, in this case the condition rankC/a = rankXco(a+) = n is automatically satisfied and we get from Corollary 2.25 the following. This result is also new even in this special setting. Corollary 2.26. Assume that (1.3) holds with X = [a,oo) and system (H) is completely controllable on [a, oo) and nonoscillatory. Then the principal solutions Ya and Yx, have the same number of focal points in the open interval (a, oo), i.e., ma(a, oo) = m-oo(a, oo). (2.94) The above results from the singular Sturmian theory of system (H) on an unbounded interval allow to describe asymptotic properties of the comparative indices fj,(Y(t),Y(t)) and fj,*(Y(t),Y(t)) when t —> oo for a pair of conjoined bases Y and Y of (H), see [86, Theorems 6.1 and 6.4] or Appendix D. This result corresponds to Theorem 2.10. Theorem 2.27. Assume that (1.3) holds with I = [a, oo) and system (H) is nonoscillatory. Then for any two conjoined bases Y and Y of (H) the limits of the comparative indices fi(Y(t),Y(t)) and /i* (Y (t), Y(/)) fort —> oo exist and lim fi(Y(t), Y(t)) = ^(Coo, C^) - mL(oo) + mL(oo), (2.95) lim »*(Y(t),Y(t)) =//(Coo, Coo), (2.96) t—>oo where Coo and Coo are the constant matrices in (2.75) corresponding to Y and Y. In the next theorem we present a formula for calculating the multiplicity of the focal point at infinity of a conjoined basis Y of system (H) in terms of the comparative index of Y with the principal solution Yt, respectively in terms of their representing matrices Coo and C^ in (2.75). In addition, we provide an interesting representation of the maximal order of abnormality o, C^) • This result corresponds to Theorem 2.11. Theorem 2.28. Assume that (1.3) holds with I = [a, oo) and system (H) is nonoscillatory. Then for any conjoined basis Y of (H) the limits of the comparative indices /((Cx^C^) and /^(Coo^C^) for i -> oo exist and lim M(Coo, Cio) = mL(oo), (2.97) t—¥oo lim n*(Coo, Co,) = n ~ doo, (2.98) t—>oo where Coo and C^ are the constant matrices in (2.75) corresponding to Y and Yt. In the last part of this subsection we compare the above results with the limiting cases of the results in Subsection 2.2.1 when we assume that system (H) is nonoscillatory and the right endpoint 28 Chapter 2. Sturmian theory b —> oo. Equations (2.43)-(2.44) in Theorem 2.3 yield that for any conjoined bases Y and Y there exist the limits lim fj,(Y(t), Y(t)) = mL(a, oo) - rhL(a, oo) + fJ,(Y(a), Y(a)), (2.99) lim = //(Y(a), F(a)) - mfl[a, oo) + rhR[a, oo), (2.100) t—>oo Then the result in Theorem 2.27 shows, how these two limits can be evaluated explicitly without using the numbers of left proper focal points of Y and Y in (a,oo) or without using the numbers of right proper focal points of Y and Y in [a, oo). Moreover, the results in Theorem 2.20 yield interesting connections with the limits of the corresponding equalities in (2.49). First of all, it is not at all clear whether the limits lim mi,b(a, b], lim mjy,[a, b) (2.101) b—>oo b—>oo exist, and if they exist, then what are their values. Below we show that both of these limits indeed exist and that the first one is equal to m,i00(a, oo] as we would formally expect, but surprisingly the second one is not equal to mRoo[a, oo) in general. More precisely, we have r ( hi (2'49) T r i\ r \ (2-84) hmmLb(a,b\ = hm mRa[a, b) = mRa[a, oo) = mLoo(a, oo], b—>oo b—>oo lim mRb[a, b) (2=9) lim mLa(a, b] = mLa(a, oo) (2=4) mRoo[a, oo) - mLa(oo). b—>oo b—>oo The above calculation shows that the second limit in (2.101) is equal to the formally expected value mRoo[a, oo) only when m,£a(oo) = 0, i.e., only when the principal solution Ya is antiprincipal at oo according to Definition 2.14 and the subsequent comments. Therefore, by taking the limit as b —> oo in the estimates in (2.50) for the left proper focal points we obtain the statement in (2.88). On the other hand, by taking the limit as b —> oo in the estimates in (2.51) for the right proper focal points we obtain the estimates mRQO[a, oo) - mLa(oo) < mR[a, oo) < mRa[a, oo). (2.102) In (2.102), the upper bound is optimal according to (2.87), while the lower bound is not in general optimal. More precisely, the lower bound in (2.102) is optimal if and only if m,£a(oo) = 0. 2.3.2. Singular Sturmian comparison theorems. In this is subsection we study a more general situation in the singular Sturmian theory on the unbounded interval X = [a, oo). More precisely, we present the Sturmian comparison theorems for conjoined bases of two nonoscillatory systems (H) and (H), which satisfy the Sturmian majorant condition (2.17). These results were developed in [87, Sections 4-6], see Appendix E. We note that the results in this subsection hold also with the left endpoint a = — oo for the interval X = (—oo, b], if we consider the corresponding concept of the minimal principal solution of (H) at minus infinity. A detailed analysis of this case is presented in [87, Section 6], see Appendix E. The situation for the Sturmian comparison theorems on the unbounded interval X = (—00,00) is slightly different and we comment on this case at the end of this subsection. First we recall a comparison result for nonoscillatory systems (H) and (H) under the majorant condition (2.17), as well as the invariance of the nonoscillation for system (H) and the transformed system (H). The latter result is based on the generalized reciprocity principle in [35, Theorem 2.2]. Proposition 2.29. Assume that (2.17) and (2.18) hold onl = [a, 00) and that system (H) is nonoscillatory. Then (i) system (H) is nonoscillatory, and (ii) for every symplectic fundamental matrix Z of (H) the transformed system (H) with the coefficient matrix ~H(t) given in (2.70) is nonoscillatory. We denote by Yx>, Yco, Xx> the minimal principal solutions of nonoscillatory systems (H), (H), (H) at infinity, respectively. Moreover, let Z^, Z^, Z^ be the associated symplectic fundamental matrices defined in (2.74). In the remaining part of this subsection we will consider the transformed system Section 2.3. Singular Sturmian theory 29 (H) with respect to the transformation matrix Z^. The following result shows that under natural assumptions the minimal principal solution Yx of (H) at infinity is transformed into the minimal principal solution Yx of (H) at infinity and vice versa. Theorem 2.30. Assume that (2.17) and (2.18) hold onI= [a,oo) and that system (H) is nonoscil-latory. Then the conjoined basis Z^Yx is the minimal principal solution of (H) at infinity and the conjoined basis ZooYx is the minimal principal solution of (H) at infinity. That is, the equalities Z^-Yoo = YoK and Z^Y^ = Y^K-1 hold for some constant invertible n X n matrix K. The next result provides the singular Sturmian comparison theorem for two systems (H) and (H) on the unbounded interval X = [a, oo) satisfying the majorant condition (2.17). In particular, it generalizes the comparison formulas in Theorems 2.12 and 2.13, as well as the separation formulas in Theorem 2.17 to this case. Theorem 2.31 (Singular Sturmian comparison theorem). Assume that (2.17) and (2.18) hold on X = [a, oo) and that system (H) is nonosdilatory. Then for every conjoined basis Y of (H) and for every conjoined basis Y of (H) we have mL(a, oo] - mL(a, oo] = mL(a, oo] + n(Y(a), Yoo(a)) - n(Y(a), Yoc(a)), (2.103) mR[a,oo) - mR[a, oo) = mR[a, oo) + fi*(Y(a), Foo(a)) - fi*(Y(a), Foo(a)), (2.104) where m£,(a, oo] and fhR[a, oo) are the numbers of left and right proper focal points of the conjoined basis Y := Z^Y of (H) in the indicated intervals. In the following we present the special case of Theorem 2.31 for Y := Y^. It is a generalization of estimate (2.26) to the uncontrollable systems (H) and (H). At the same time it is an extensions of (2.67) and (2.69) to the case of b = oo with the fundamental matrix Z(t) := Zoo(t). Theorem 2.32 (Singular Sturmian comparison theorem). Assume that (2.17) and (2.18) hold on X = [a, oo) and that system (H) is nonos dilatory. Then for every conjoined basis Y of (H) we have mL(a, oo] = fhLoo(a, oo] + mL(a, oo] - fi{Y(a), Foo(a)), (2.105) mR[a, oo) = rhRoo[a, oo) + mR[a, oo) + n*(Y(a), Yoc(a)), (2.106) where rh^ia, oo] and fhR[a, oo) are the numbers of left and right proper focal points of the conjoined basis Y := Z^Y of (H) in the indicated intervals. The results in Theorem 2.31 (or Theorem 2.32) allow to derive various estimates for the numbers of left and right proper focal points of conjoined bases of (H) and (H). In particular, in the next result we show the exact relationship between the numbers of proper focal points of the (minimal) principal solutions Yoo, Foo, Yx and Ya, Ya, Ya of systems (H), (H), (H). Theorem 2.33. Assume that (2.17) and (2.18) hold onI= [a,oo) and that system (H) is nonoscil- latory. Then we have mLQO(a, oo] = fhLQO(a, oo] + mLQO(a, oo] - ^(Y^a), Y^a)), (2.107) mRQO[a, oo) = fhRoo[a, oo) +fhRoo[a, oo) + //(^(a), Foo(a)), (2.108) mLa(a, oo] = rhLa(a, oo] + mLa(a, oo] + Ju*(lcx)(a), Yoc(a)), (2.109) mRa[a, oo) = fhRa[a, oo) + mRa[a, oo) - ^(^(a), Foo(a)), (2.110) The following result confirms the intuitively expected fact that, given the same initial conditions, conjoined bases of the majorant system (H) have in general more focal points than conjoined bases of the minorant system (H). It is a generalization of the first part of [58, Corollary 7.3.2, pg. 196]. 30 Chapter 2. Sturmian theory Theorem 2.34 (Singular Sturmian comparison theorem). Assume that (2.17) and (2.18) hold on X = [a,oo) and that system (H) is nonosdilatory. Let Y and Y be conjoined bases of (H) and (H), respectively, such that Y(a) = Y(a) K for some invertible matrix K. Then m£,(a, oo] — fh^ia, oo] = fh^ia, oo] > 0, (2.111) mR[a, oo) — mR[a, oo) = fhR[a, oo) > 0, (2.112) where mi(a, oo] and mR[a, oo) are i/ie numbers of left and right focal points of the conjoined basis Y := Z^-Y of (H) in the indicated intervals. Next we compare the numbers of focal points of the (minimal) principal solutions Yx,, Y^, Yx and Ya, Ya, Ya. We also provide universal lower and upper bounds for the numbers of focal points of conjoined bases of (H) and (H). Theorem 2.35. Assume that (2.17) and (2.18) hold onI= [a,oo) and that system (H) is nonoscil-latory. Then for every conjoined basis Y of (H) the estimates in (2.86) and (2.87) hold, where the lower and upper bounds satisfy fhLa(a, oo] + mLa(a, oo] < mLa(a, oo], mLoo(a, oo] < fhLoo(a, oo] + fhLoo(a, oo], (2.113) mRQO[a,oo) + fhRoo[a,oo) < mRoo[a, oo), mRa[a, oo) < fhRa[a, oo) + mRa[a, oo). (2.114) Moreover, for every conjoined basis Y of (H) we have the estimates mLa(a, oo] < fhL(a, oo] < fhLoo(a, oo] < mLoo(a, oo], (2.115) mRQO[a,oo) < fhR[a, oo) < fhRa[a, oo) < mRa[a, oo). (2.116) If assumptions (2.17) and (2.18) hold on the interval X = (—00,00) and system (H) is nonoscillatory, then for every conjoined basis Y of (H) we have the estimates mL(-oo, 00] < mLoo(-oo, 00] < mLoo(-oo, 00] + mLoo(-oo, 00], (2.117) mR(-oo, 00) > mRQO(-oo, 00) > fhRoo(-oo, 00) + mRQO(-oo, 00). (2.118) The inequalities in (2.117) follow from the singular Sturmian separation theorem (i.e., from (2.86) in Theorem 2.21) for system (H) with a —> —00 and from (2.107) with a —> —00 by dropping the last term with the comparative index. The inequalities in (2.118) follow from the singular Sturmian separation theorem (i.e., from (2.87) in Theorem 2.21) for system (H) with a —> —00 and from (2.108) with a —> —00 by dropping the last term with the dual comparative index. The lower bounds in (2.118) for the numbers of right proper focal points of Y in the interval (—00, 00) improve the lower bound m00(—00,00) obtained in (2.27) by Došlý and Kratz. The final result in this subsection describes properties of the comparative indices ju(F(i), Y^it)) and fi*(Y(t), Y^it)) when t —> 00 for a conjoined bases Y of (H), see [87, Theorems 4.6]. Theorem 2.36. Assume that (2.17) and (2.18) hold onI= [a, 00) and that system (H) is nonoscillatory. Then for every conjoined basis Y of (H) the limits lim íi(Y(t),Ýco(t)) and lim fi*(Y(t), Ý^t)) (2.119) exist and satisfy lim /i(y(í), Ýoo(t)) = mL(oo) - mL(oo) + mLoo(oo), (2.120) t—>oo Um//*(y(i),Y'oo(i)) = 0, (2.121) t—>oo where mi(oo) is the multiplicity of a focal point at infinity of the conjoined basis Y := Z^Y of the transformed system (H). We remark that equation (2.120) is an extension of formula (2.66) in Theorem 2.12 with Y := Yto to the case to = oo. At the same time it is a generalization of formula (2.95) in Theorem 2.27 with Y := Yo to two systems (H) and (H) satisfying (2.17). Note that equation (2.121) represents Section 2.3. Singular Sturmian theory 31 an extension to to = 00 of the left continuity of the dual comparative index fi*(Y(t),Yt0(t)) at the point to m Theorem 2.13. In the last part of this subsection we comment on the above results in connection with the results in Subsection 2.2.2, when we assume that system (H) is nonoscillatory and the right endpoint b —> 00. Equations (2.67) and (2.67) in Theorems 2.12 and 2.13 yield that for any conjoined bases Y and Y of systems (H) and (H), respectively, there exist the limits lim fJ,(Y(t), Y(t)) = mL(a, 00) - rhL(a, 00) - rhL(a, 00) + fj,(Y(a),Y(a)), (2.122) t—>oo lim /i*(y(t), Y(t)) = fi*{Y{a), Y(a)) — mjj[a, 00) + mR[a, 00) + mR[a, 00), (2.123) t—>oo where rhiia, 00) and mR[a, 00) are the numbers of left and right proper focal points of the conjoined basis Y := Z^Y of system (H) in the indicated intervals. Note that these numbers are finite according to Proposition 2.29. The result in Theorem 2.36 then shows that in the special case of Y = Foo these two limits can be evaluated explicitly. When dealing with the singular Sturmian separation theorems on the interval X = (—00,00) in Subsection 2.3.1, we added together the corresponding Sturmian separation theorems on X = [a, 00) and on X = (—00, a] to get the results on the interval X = (—00,00). The situation for the Sturmian comparison theorems onX = (—00,00) is different, since the results on the interval X = [a, 00) in this subsection together with the corresponding results on the interval X = (—00, a] in general do not combine to obtain the Sturmian comparison theorems on the entire interval X = (—00, 00). The main reason can be seen in Theorem 2.31, which implies that we need to employ two different transformation matrices Z±oc(t) in the neighborhoods of ±00. Hence, we obtain two different transformation systems (H) - one on the interval X = [a, 00) with the transformation matrix Zoo(t) and one on the interval X = (—00, a] with the transformation matrix Z-oo(t). Nevertheless, the results presented in this subsection remain valid also on the interval X = (—00,00) under the additional assumption that the minimal principal solutions Y^ and Y_oo at ±00 of the minorant system (H) coincide, meaning that Y_oo is a constant nonsingular multiple of Y^ (which yields that Z^t) = Z-^t)). 2.3.3. Sturmian comparison theorems for controllable systems. In this subsection we discuss singular Sturmian comparison theorems for two linear Hamiltonian systems (H) and (H), satisfying the standard majorant condition (2.17) and the Legendre condition (2.18), in the the special setting. Namely, we consider the situation when • the interval X = (a, 00) is open, where the point a can be singular (including the case of a = —00), • both systems (H) and (H) are completely controllable on the interval (a, 00) and nonoscillatory (at a and at infinity), • the principal solution of the minorant system (H) at a is at the same time principal at infinity, i.e., Ya = holds. The motivation of this kind of results comes from a singular Sturmian comparison theorem for the second order Sturm-Liouville differential equations, which was proven in [1,2] by Aharonov and Elias. More precisely, the result in [1, Theorem 1] or [2, Theorem 1] states that under a certain strict majorant condition on the coefficients every nontrivial solution of the majorant equation has a zero in the open interval (a, 00), if the principal solution at a of the minorant equation is principal also at infinity. This corresponds, in the spirit of Remark 2.16, to the situation when the principal solution of the minorant equation has zeros at a and at infinity, and hence the results in [1, Theorem 1] or [2, Theorem 1] can be interpreted from this point of view as the standard Sturmian comparison theorems on a compact interval [a, b]. Comparing with Subsection 2.3.2 now we will consider the endpoint a £ RU{-00} to be singular, especially we allow a = —00. However, the results of this subsection are formulated in a way which includes both regular and singular endpoint a. For this purpose we will utilize the definition of the principal solution Ya of (H) at a as a conjoined basis of (H) with the matrix Xa(t) invertible on (a, j3\ 32 Chapter 2. Sturmian theory for some (3 £ (a, oo) and satisfying \ua ^-j\-1(s)B(s)X^-1(s)ds^j = 0. (2.124) This unified definition (regarding with the definition of the principal solution at ±oo) is justified by the comments in [3, pg. 173] and by the results of [85, Theorem 5.8] and in Subsections 2.3.1 and 2.3.2. More precisely, the principal solution of (H) at a defined according to (2.124) coincides with the principal solution of (H) at the point a defined in (1.12), i.e., by the initial condition Ya(a) = E, when the endpoint a is regular. Following Theorem 2.15, for a conjoined basis Y of a nonoscillatory and completely controllable system (H) the multiplicity of its focal point at a and at oo is defined by the quantity m(t0) :=def W(Yto,Y) = n - r ma(a, oo) = rha(a, oo) + fhRa[a, oo). (2.128) As a consequence of Theorem 2.37 we obtain that conjoined bases of the majorant system (H), except of the principal solution Ya of (H) at a (up to a right constant nonsingular multiple), have at least one focal point in the open interval (a, oo). This represents a direct generalization of the Sturmian comparison theorems in [1, Theorem 1] and [2, Theorem 1] by Aharonov and Elias. Corollary 2.38. Assume that (2.17) and (2.18) hold on (a,oo) and that systems (H) and (H) are nonoscillatory and completely controllable. If Ya = Y^ and Y is a conjoined basis of (H) such that Y ^Ya, then m(a, oo) > 1. Remark 2.39. The equality in (2.128) yields the traditional comparison theorem for principal solutions Ya and Ya. Namely, if Ya has no focal points in (a, oo), then also Ya has no focal points in (a, oo). Moreover, compared to inequality (2.26) the estimate in (2.128) provides an improved lower bound for the number of focal points in (a, oo) of any conjoined basis Y of (H) for the case when Ya = Y^. In the following result we show that the number of focal points in (a, oo) of any conjoined basis Y of (H) is given by the rank of the Wronskian of Y with Ya, when the principal solution Ya of system (H) has no focal points in (a, oo). Theorem 2.40. Assume that (2.17) and (2.18) hold on (a,oo) and that systems (H) and (H) are nonoscillatory and completely controllable. In addition, assume that Ya = Y^ and that Ya has no focal points in (a, oo). Then Ya = Yx and for every conjoined basis Y of (H) we have the equalities m(a, oo) = rank W(Ya,Y), m{a) = def W(Ya, Y) = m(oo). (2.129) Section 2.4. Further research directions 33 Under the assumptions of Theorem 2.40, the equalities in (2.129) can be combined to obtain m[a, oo) = n = m(a, oo] for every conjoined basis Y of (H). We note that the same conclusion m[a, oo) =n = in (a, oo] for every conjoined basis Y of (H) follows from the singular Sturmian separation theorem (Theorem 2.21) applied to the minorant system (H) under the additional assumption Ya = Y^. Also, given that Ya = Y^ and Ya = Xx,, it follows by the transformation result in Theorem 2.30 that the statement of Theorem 2.40 can be supplemented by the additional conclusion Ya = ^oo- Note that when the point a is regular, then in (2.129) we have m(a, oo) = rankA(a) and m{a) = def X(a), since in this case W(l^, Y) = —X{a). In the second main result of this subsection we complement the result in Theorem 2.37 (resp. in Remark 2.39) by providing an exact relationship between the considered properties of the principal solutions Ya and Ya. This relationship is expressed in terms of the Riccati quotient associated with the principal solution Ya of (H). Theorem 2.41 (Singular Sturmian comparison theorem). Assume that (2.17) and (2.18) hold on (a,oo) and that systems (H) and (H) are nonosdilatory and completely controllable. In addition, assume that Ya = holds. Then the following statements are equivalent. (i) The principal solution Ya of (H) has no focal points in (a,oo). (ii) The principal solution Ya of (H) has no focal points in (a, oo) and H(t) - H(t) = ( — Qa(t), if [B(t) - B(t)} ( - Qa(t), I), t G (a, oo), (2.130) where Qa(t) := Ua(i) A~1(i) is the symmetric Riccati quotient which corresponds to the principal solution Ya of (H) on (a, oo). In this case, the symmetric Riccati quotient Qa(t) := ř7a(í)A~1(í) corresponding to the principal solution Ya of (H) satisfies Qa(t) = Qa(t) on (a, oo). The above results in Theorems 2.37, 2.40, and 2.41 are new even for the second order Sturm-Liouville differential equations. And even in this special case they generalize the Sturmian comparison theorems in [1, Theorem 1] and [2, Theorem 1] by Aharonov and Elias. More details on this subject are presented in [88, Section 4]. 2.4. Further research directions The research in the theory of Riccati matrix differential equations can be pursued in several directions. One possibility is to consider the discrete analogy, i.e., the Riccati matrix difference equations. These are important objects e.g. for discrete variational analysis or discrete filtering theory [5-7,18], as well as for the discrete oscillation theory [15,16,27,31,97] or for numerical algorithms for computing the eigenvalues of symmetric banded matrices [61]. An interesting order preserving property of the discrete Riccati matrix equation was recently derived in [95] by Stoudková Růžičková, which is an analogue of the known order preserving property of the Riccati matrix differential equation (R), see the work by Stokes in [74,75]. Open problems in the Sturmian theory of linear Hamiltonian systems include the development of this theory when we remove the Legendre condition (1.3), thus considering the oscillation numbers for conjoined bases of system (H) as it is presented in [39] by Elyseeva. This naturally leads to the connections between the Sturmian theory and the theory of Maslov index [10,40,51,52]. In a close relationship with the latter research direction we mention the classical Gelfand-Lidskii-Yakubovich oscillation theory [44,62,98-100], to which we recently contributed by providing an explicit connection between the Lidskii angles of symplectic matrices with the comparative index [90]. Another open problem is to find explicit expressions for the limits of the comparative index in (2.122) and (2.123) for arbitrary conjoined bases Y of system (H) and Y of system (H). Moreover, the question of the validity of the Sturmian comparison theorems for systems (H) and (H) on the unbounded interval X = (—oo, oo) remains an open problem when the minimal principal solutions Y^ and Y_oo of (H) at ±oo differ, meaning that Y_oo is not a constant nonsingular multiple of Y^. In the spectral theory of linear Hamiltonian systems we may consider singular oscillation theorems on unbounded intervals [45] 34 Chapter 2. Sturmian theory or applications of the theory of principal and antiprincipal solutions of (H) at infinity in the Weyl-Titchmarsh theory [94]. In general, we can say that it is indeed beneficial to develop the oscillation and spectral theory of linear Hamiltonian systems together with the oscillation and spectral theory of their discrete time counterparts, which are the symplectic difference systems - see the recent monograph ['. '] on this subject and the numerous references therein. This approach can be highlighted by developing the Sturmian theory for symplectic (or Hamiltonian) dynamic systems on time scales, such as in the references [4,17, 28,93,101]. In particular, it is an open problem what is the multiplicity of a focal point of a conjoined basis in this abstract setting. Declaration about co-authorship Hereby I declare that the results in papers [84,86,87], which are contained in Appendices C-E of this habilitation thesis, were obtained in a joint co-authorship with prof. Roman Simon Hilscher. The contribution of both authors in these publications was equal. In Brno, May 21, 2021 Peter Sepitka 35 References [1] D. Aharonov, U. Elias, Singular Sturm comparison theorems, J. Math. Anal. Appl. 371 (2010), no. 2, 759-763. [2] D. Aharonov, U. Elias, Sturm comparison theorem in singular situations, Fund. Differ. Equ. 18 (2011), no. 3-4, 171-175. [3] C. D. Ahlbrandt, Principal and antiprincipal solutions of self-adjoint differential systems and their reciprocals, Rocky Mountain J. Math. 2 (1972), 169-182. [4] C. D. Ahlbrandt, M. Bohner, J. Ridenhour, Hamiltonian systems on time scales, J. Math. Anal. Appl. 250 (2000), no. 2, 561-578. [5] C. D. Ahlbrandt, M. Heifetz, Discrete Riccati equations of filtering and control, in: "Proceedings of the First International Conference on Difference Equations" (San Antonio, TX, 1994), S. Elaydi, J. Graef, G. Ladas, and A. Peterson, editors, pp. 1-16, Gordon and Breach, Newark, NJ, 1996. [6] C. D. Ahlbrandt, M. Heifetz, J. W. Hooker, W. T. Patula, Asymptotics of discrete time Riccati equations, robust control, and discrete linear Hamiltonian systems, PanAmer. Math. J. 5 (1996), no. 2, 1-39. [7] C. D. Ahlbrandt, A. C. Peterson, Discrete Hamiltonian Systems. Difference Equations, Continued Fractions, and Riccati Equations, Kluwer Texts in the Mathematical Sciences, Vol. 16, Kluwer Academic Publishers Group, Dordrecht, 1996. [8] J. Allwright, R. Vinter, Second order conditions for periodic optimal control problems, Control Cybernet. 34 (2005), no. 3, 617-643. [9] W. O. Amrein, A. M. Hinz, D. B. Pearson, Sturm-Liouville theory. Past and present, Including papers from the International Colloquium held at the University of Geneva, (Geneva, 2003), Birkhauser Verlag, Basel, 2005. [10] V. I. Arnold, Sturm theorems and symplectic geometry, Funktsional. Anal, i Prilozhen. 19 (1985), no. 4, 1-10; translated in Functional Anal. Appl. 19 (1985), no. 4, 251-259. [11] F. V. Atkinson, Discrete and Continuous Boundary Problems, Academic Press, New York - London, 1964. [12] R. Bellman, Introduction to the mathematical theory of control processes. Vol. I: Linear equations and quadratic criteria, Mathematics in Science and Engineering, Vol. 40, Academic Press, New York, NY, 1967. [13] A. Ben-Israel, T. N. E. Greville, Generalized Inverses: Theory and Applications, Second Edition, Springer-Verlag, New York, NY, 2003. [14] D. S. Bernstein, Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory, Princeton University Press, Princeton, 2005. [15] M. Bohner, Riccati matrix difference equations and linear Hamiltonian difference systems, Dynam. Contin. Discrete Impuls. Systems 2 (1996), no. 2, 147-159. [16] M. Bohner, O. Došlý, Disconjugacy and transformations for symplectic systems, Rocky Mountain J. Math. 27 (1997), no. 3, 707-743. [17] M. Bohner, O. Došlý, R. Hilscher, Linear Hamiltonian dynamic systems on time scales: Sturmian property of the principal solution, Nonlinear Anal. 47 (2001), no. 2, 849-860. [18] L. Bourdin, E. Trélat, Unified Riccati theory for optimal permanent and sampled-data control problems in finite and infinite time horizons, SIAM J. Control Optim. 59 (2021), no. 1, 489-508. [19] S. L. Campbell, C. D. Meyer, Generalized Inverses of linear Transformations, Reprint of the 1991 corrected reprint of the 1979 original, Classics in Applied Mathematics, Vol. 56, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2009. [20] W. A. Coppel, Comparison theorems for canonical systems of differential equations, J. Math. Anal. Appl. 12 (1965), no. 2, 306-315. [21] W. A. Coppel, Disconjugacy, Lecture Notes in Mathematics, Vol. 220, Springer-Verlag, Berlin - New York, 1971. [22] Z. Došlá, O. Došlý, Quadratic functionals with general boundary conditions, Appl. Math. Optim. 36 (1997), 243-262. [23] O. Došlý, Transformations of linear Hamiltonian systems preserving oscillatory behaviour, Arch. Math. (Brno) 27b (1991), 211-219. [24] O. Došlý, Principal solutions and transformations of linear Hamiltonian systems, Arch. Math. (Brno) 28 (1992), 113-120. [25] O. Došlý, Transformations and oscillatory properties of linear Hamiltonian systems - continuous versus discrete, in: "Equadiff 9", Proceedings of the Conference on Differential Equations and their Applications (Brno, 1997), R. P. Agarwal, F. Neuman, J. Vosmanský, editors, pp. 49-61, Masaryk University, Brno, 1998. 37 38 References [26] O. Došlý, Relative oscillation of linear Hamiltonian differential systems, Math. Nachr. 290 (2017), no. 14-15, 2234-2246. [27] O. Došlý, J. V. Elyseeva, R. Simon Hilscher, Symplectic Difference Systems: Oscillation and Spectral Theory, Pathways in Mathematics, Birkhauser/Springer, Cham, 2019. [28] O. Došlý, R. Hilscher, Disconjugacy, transformations and quadratic functionals for symplectic dynamic systems on time scales, J. Differ. Equations Appl. 7 (2001), 265-295. [29] O. Došlý, W. Kratz, Singular Sturmian theory for linear Hamiltonian differential systems, Appl. Math. Lett. 26 (2013), no. 12, 1187-1191. [30] U. Elias, Oscillation Theory of Two-term Differential Equations, Mathematics and its Applications, Vol. 396. Kluwer Academic Publishers Group, Dordrecht, 1997. [31] J. V. Elyseeva, An algorithm for solving the matrix difference Riccati equation, Zh. Vychisl. Mat. Mat. Fiz. 39 (1999), no. 2, 187-194, translation in Comput. Math. Math. Phys. 39 (1999), no. 2, 177-184. [32] J. V. Elyseeva, The comparative index for conjoined bases of symplectic difference systems, in: "Difference Equations, Special Functions, and Orthogonal Polynomials" (Munich, 2005), pp. 168-177, World Scientific, London, 2007. [33] J. V. Elyseeva, Comparative index for solutions of symplectic difference systems, Differential Equations 45 (2009), no. 3, 445-459. [34] J. Elyseeva, Comparison theorems for conjoined bases of linear Hamiltonian differential systems and the comparative index, J. Math. Anal. Appl. 444 (2016), no. 2, 1260-1273. [35] J. V. Elyseeva, On symplectic transformations of linear Hamiltonian differential systems without normality, Appl. Math. Lett. 68 (2017), 33-39. [36] J. V. Elyseeva, The comparative index and transformations of linear Hamiltonian differential systems, Appl. Math. Comput. 330 (2018), 185-200. [37] J. V. Elyseeva, Oscillation theorems for linear Hamiltonian systems with nonlinear dependence on the spectral parameter and the comparative index, Appl. Math. Lett. 90 (2019), 15-22. [38] J. V. Elyseeva, Relative oscillation of linear Hamiltonian differential systems without monotonicity, Appl. Math. Lett. 103 (2020), Article 106173, 8 pp. [39] J. V. Elyseeva, Comparison theorems for conjoined bases of linear Hamiltonian systems without monotonicity, Monatsh. Math. 193 (2020), no. 2, 305-328. [40] J. Elyseeva, P. Sepitka, R. Simon Hilscher, Oscillation numbers for Lagrangian paths and Maslov index, submitted (2021). [41] R. Fabbri, R. Johnson, S. Novo, C. Nunez, Some remarks concerning weakly disconjugate linear Hamiltonian systems, J. Math. Anal. Appl. 380 (2011), no. 2, 853-864. [42] R. Fabbri, R. Johnson, C. Nunez, On the Yakubovich frequency theorem for linear non-autonomous control processes, Discrete Contin. Dyn. Syst. 9 (2003), no. 3, 677-704. [43] R. Fabbri, C. Nunez, On the solvability of the Yakubovich linear-quadratic infinite horizon minimization problem, Ann. Mat. Pura Appl. (4) 199 (2020), no. 5, 1713-1735. [44] I. M. Gelfand, V. B. Lidskii, On the structure of the regions of stability of linear canonical systems of differential equations with periodic coefficients, in: "Twelve Papers on Function Theory, Probability and Differential Equations", American Mathematical Society Translations, Ser. 2, Vol. 8, pp. 143-181, American Mathematical Society, Providence, RI, 1958. [45] F. Gesztesy, M. Zinchenko, Renormalized oscillation theory for Hamiltonian systems, Adv. Math. 311 (2017), 569-597. [46] P. Hartman, Ordinary Differential Equations, John Wiley, New York, 1964. [47] M. R. Hestenes, Quadratic control problems, J. Optim. Theory Appl. 17 (1975), no. 1-2, 1-42. [48] R. Hilscher, V. Zeidan, Applications of time scale symplectic systems without normality, J. Math. Anal. Appl. 340 (2008), no. 1, 451-465. [49] R. Hilscher, V. Zeidan, Time scale embedding theorem and coercivity of quadratic functionals, Analysis (Munich) 28 (2008), no. 1, 1-28. [50] R. Hilscher, V. Zeidan, Riccati equations for abnormal time scale quadratic functionals, J. Differential Equations 244 (2008), no. 6, 1410-1447. [51] P. Howard, S. Jung, B. Kwon, The Maslov index and spectral counts for linear Hamiltonian systems on [0,1], J. Dynam. Differential Equations 30 (2018), no. 4, 1703-1729. [52] P. Howard, Y. Latushkin, A. Sukhtayev, The Maslov index for Lagrangian pairs on K2n, J. Math. Anal. Appl. 451 (2017), no. 2, 794-821. [53] R. Johnson, S. Novo, C. Núfiez, R. Obaya, Uniform weak disconjugacy and principal solutions for linear Hamiltonian systems, in: 'Proceedings of the International Conference on Delay Differential and Difference Equations and Applications" (Balatonfuered, Hungary, 2013), Springer, Berlin, 2014. [54] R. Johnson, S. Novo, C. Núfiez, R. Obaya, Nonautonomous linear-quadratic dissipative control processes without uniform null controllability, J. Dynam. Differential Equations 29 (2017), no. 2, 355-383. [55] R. Johnson, C. Núfiez, R. Obaya, Dynamical methods for linear Hamiltonian systems with applications to control processes, J. Dynam. Differential Equations 25 (2013), no. 3, 679-713. References 39 [56] R. Johnson, R. Obaya, S. Novo, C. Nunez, R. Fabbri, Nonautonomous Linear Hamiltonian Systems: Oscillation, Spectral Theory and Control, Developments in Mathematics, Vol. 36, Springer, Cham, 2016. [57] R. E. Kalman, R. S. Bucy, New results in linear filtering and prediction theory, Trans. ASME Ser. D J. Basic Engrg. 83 (1961), 95-108. [58] W. Kratz, Quadratic Functionals in Variational Analysis and Control Theory, Akademie Verlag, Berlin, 1995. [59] W. Kratz, Definiteness of quadratic functionals, Analysis 23 (2003), no. 2, 163-183. [60] W. Kratz, R. Simon Hilscher, Rayleigh principle for linear Hamiltonian systems without controllability, ESAIM Control Optim. Calc. Var. 18 (2012), no. 2, 501-519. [61] W. Kratz, M. Tentler, Recursion formulae for the characteristic polynomial of symmetric banded matrices, Linear Algebra Appl. 428 (2008), 2482-2500. [62] V. B. Lidskii, Oscillation theorems for canonical systems of differential equations, (Russian) Dokl. Akad. Nauk SSSR (N.S.) 102 (1955), no. 5, 877-880. Translation in: NASA Technical Translation, TT F-14, 696 (1973), 9 pp. [63] B. P. Molinari, Nonnegativity of a quadratic functional, SIAM J. Control 13 (1975), no. 4, 792-806. [64] C. Núfiez, R. Obaya, Non-Atkinson perturbations of nonautonomous linear Hamiltonian systems: Exponential dichotomy and nonoscillation, J. Dynam. Differential Equations 31 (2019), no. 3, 1397-1426. [65] W. T. Reid, Riccati matrix differential equations and non-oscillation criteria for associated linear differential systems, Pacific J. Math. 13 (1963), 665-685. [66] W. T. Reid, Principal solutions of nonoscillatory linear differential systems, J. Math. Anal. Appl. 9 (1964), 397-423. [67] W. T. Reid, Ordinary Differential Equations, John Wiley fe Sons, Inc., New York - London - Sydney, 1971. [68] W. T. Reid, Riccati Differential Equations, Academic Press, New York - London, 1972. [69] W. T. Reid, Sturmian Theory for Ordinary Differential Equations, Springer-Verlag, New York - Berlin - Heidelberg, 1980. [70] F. S. Rofe-Beketov, A. M. Kholkin, Spectral Analysis of Differential Operators, Translated from the Russian by Ognjen Milatovic and revised by the authors. With a foreword by Vladimir A. Marchenko. World Scientific Monograph Series in Mathematics, Vol. 7, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2005. [71] V. Růžičková, Discrete Symplectic Systems and Definiteness of Quadratic Functionals, PhD dissertation, Masaryk University, Brno, 2006. Available at https: //is .muni. cz/th/p9iz7/?lang=en. [72] J. L. Speyer, D. H. Jacobson, Primer on Optimal Control Theory, Advances in Design and Control, Vol. 20, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2010. [73] G. Stefani, P. Zezza, Constrained regular LQ-control problems, SIAM J. Control Optim. 35 (1997), no. 3, 876-900. [74] A. N. Stokes, Differential Inequalities and the Matrix Riccati Equation, PhD thesis, Australian National University, Canberra, 1972, see also the abstract in Bull. Austral. Math. Soc. 9 (1973), no. 2, 315-317. [75] A. N. Stokes, A special property of the matrix Riccati equation, Bull. Austral. Math. Soc. 10 (1974), no. 2, 245-253. [76] P. Sepitka, Theory of Principal Solutions at Infinity for Linear Hamiltonian Systems, PhD dissertation, Masaryk University, Brno, 2014. Available at https: //is .muni. cz/th/vqad7/?lang=en. [77] P. Sepitka, Riccati equations for linear Hamiltonian systems without controllability condition, Discrete Contin. Dyn. Syst. 39 (2019), no. 4, 1685-1730. [78] P. Sepitka, Genera of conjoined bases for (non)oscillatory linear Hamiltonian systems: extended theory, J. Dynam. Differential Equations 32 (2020), no. 3, 1139-1155. [79] P. Sepitka, R. Simon Hilscher, Minimal principal solution at infinity for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations 26 (2014), no. 1, 57-91. [80] P. Sepitka, R. Simon Hilscher, Principal solutions at infinity of given ranks for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations 27 (2015), no. 1, 137-175. [81] P. Sepitka, R. Simon Hilscher, Principal and antiprincipal solutions at infinity of linear Hamiltonian systems, J. Differential Equations 259 (2015), no. 9, 4651-4682. [82] P. Sepitka, R. Simon Hilscher, Genera of conjoined bases of linear Hamiltonian systems and limit characterization of principal solutions at infinity, J. Differential Equations 260 (2016), no. 8, 6581-6603. [83] P. Sepitka, R. Simon Hilscher, Reid's construction of minimal principal solution at infinity for linear Hamiltonian systems, in: "Differential and Difference Equations with Applications", Proceedings of the International Conference on Differential fe Difference Equations and Applications (Amadora, 2015), S. Pinelas, Z. Došlá, O. Došlý, and P. E. Kloeden, editors, Springer Proceedings in Mathematics fe Statistics, Vol. 164, pp. 359-369, Springer, Berlin, 2016. [84] P. Sepitka, R. Simon Hilscher, Comparative index and Sturmian theory for linear Hamiltonian systems, J. Differential Equations 262 (2017), no. 2, 914-944. [85] P. Sepitka, R. Simon Hilscher, Focal points and principal solutions of linear Hamiltonian systems revisited, J. Differential Equations 264 (2018), no. 9, 5541-5576. [86] P. Sepitka, R. Simon Hilscher, Singular Sturmian separation theorems on unbounded intervals for linear Hamiltonian systems, J. Differential Equations 266 (2019), no. 11, 7481-7524. [87] P. Sepitka, R. Simon Hilscher, Singular Sturmian comparison theorems for linear Hamiltonian systems, J. Differential Equations 269 (2020), no. 4, 2920-2955. 40 References [88] P. Sepitka, R. Simon Hilscher, Sturmian comparison theorems for completely controllable linear Hamiltonian systems in singular case, J. Math. Anal. Appl. 487 (2020), no. 2, Art. 124030, 14 pp. [89] P. Sepitka, R. Simon Hilscher, Distribution and number of focal points for linear Hamiltonian systems, Linear Algebra Appl. 611 (2021), 26-45. [90] P. Sepitka, R. Simon Hilscher, Comparative index and Lidskii angles for symplectic matrices, Linear Algebra Appl. 624 (2021), 174-197. [91] R. Simon Hilscher, Sturmian theory for linear Hamiltonian systems without controllability, Math. Nachr. 284 (2011), no. 7, 831-843. [92] R. Simon Hilscher, On general Sturmian theory for abnormal linear Hamiltonian systems, in: "Dynamical Systems, Differential Equations and Applications", Proceedings of the 8th AIMS Conference on Dynamical Systems, Differential Equations and Applications (Dresden, 2010), W. Feng, Z. Feng, M. Grasselli, A. Ibragimov, X. Lu, S. Siegmund, and J. Voigt, editors, Discrete Contin. Dynam. Systems, Suppl. 2011, pp. 684-691, American Institute of Mathematical Sciences (AIMS), Springfield, MO, 2011. [93] R. Simon Hilscher, V. Zeidan, Rayleigh principle for time scale symplectic systems and applications, Electron. J. Qual. Theory Differ. Equ. 2011 (2011), no. 83, 26 pp. (electronic). [94] R. Simon Hilscher, P. Zemánek, On square integrable solutions and principal and antiprincipal solutions for linear Hamiltonian systems, Ann. Mat. Pura Appl. (4) 197 (2018), no. 1, 283-306. [95] V. Stoudková Růžičková, Discrete Riccati matrix equation and the order preserving property, Linear Algebra Appl. 618 (2021), 58-75. [96] M. Wahrheit, Eigenvalue problems and oscillation of linear Hamiltonian systems, Int. J. Difference Equ. 2 (2007), no. 2, 221-244. [97] H. K. Wimmer, M. Pavon, A comparison theorem for matrix Riccati difference equations, Systems Control Lett. 19 (1992), no. 3, 233-239. [98] V. A. Yakubovich, Arguments on the group of symplectic matrices, (Russian) Mat. Sb. (N.S.) 55 (97) (1961), no. 3, 255-280. [99] V. A. Yakubovich, Oscillatory properties of solutions of canonical equations, in: "Fifteen Papers on Differential Equations", American Mathematical Society Translations, Ser. 2, Vol. 42, pp. 247-288, American Mathematical Society, Providence, RI, 1964. [100] V. A. Yakubovich, Nonoscillation of linear periodic Hamiltonian equations and related problems, Algebra i Analiz 3 (1991), no. 5, 229-253; translation in Si. Petersburg Math. J. 3 (1992), no. 5, 1165-1188. [101] A. Zafer, The stability of linear periodic Hamiltonian systems on time scales, Appl. Math. Lett. 26 (2013), no. 3, 330-336. [102] V. Zeidan, Sufficiency criteria via focal points and via coupled points, SIAM J. Control Optim. 30 (1992), no. 1, 82-98. [103] V. Zeidan, The Riccati equation for optimal control problems with mixed state-control constraints: necessity and sufficiency, SIAM J. Control Optim. 32 (1994), no. 5, 1297-1321. [104] V. Zeidan, New second-order optimality conditions for variational problems with C2-Hamiltonians, SIAM J. Control Optim. 40 (2001), no. 2, 577-609. APPENDIX A Paper by Sepitka (JDDE 2020) This paper entitled "Genera of conjoined bases for (non)oscillatory linear Hamiltonian systems: extended theory" appeared in the Journal of Dynamics and Differential Equations, 32 (2020), no. 3, 1139-1155, see item [78] in the bibliography. 41 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 https://doi.org/! 0.1007/s10884-019-09810-w <8> Check for updates Genera of Conjoined Bases for (Non)oscillatory Linear Hamiltonian Systems: Extended Theory Peter Sepitka1© Received: 9 August 2017 / Revised: 24 October 2019 / Published online: 18 November 2019 © Springer Science+Business Media, LLC, part of Springer Nature 2019 Abstract In this paper we study the properties of conjoined bases of a general linear Hamiltonian system without any controllability condition. When the Legendre condition holds and the system is nonoscillatory, it is known from our previous work that conjoined bases with eventually the same image form a special structure called a genus. In this work we extend the theory of genera of conjoined bases to arbitrary systems, for which the Legendre condition is not assumed and/or the system may be oscillatory. We derive a classification of all genera of conjoined bases and show that they form a complete lattice. These results are based on the relationship between subspaces of solutions of a linear control system and orthogonal projectors satisfying a certain Riccati type differential equation. The presented theory is applied in our paper (Sepitka in Discrete Contin Dyn Syst 39(4): 1685-1730,2019) to general Riccati matrix differential equations for possibly uncontrollable linear Hamiltonian systems. Keywords Linear Hamiltonian system • Genus of conjoined bases • Riccati differential equation • Controllability • Orthogonal projector Mathematics Subject Classification 34C10 1 Introduction Let « G N be a given dimension and A, B, C : [a, oo) —> Wixn be given piecewise continuous matrix-valued functions such that B (?) and C (?) are symmetric. In this paper we study the properties of conjoined bases of the linear Hamiltonian system x' — A(?) x + B{t) u, u' — C{t)x - AT(t)u, te[a,oo). (H) This research was supported by the Czech Science Foundation under Grant GA16-00611S. The revised version of this paper was prepared under the support of the Grant GA19-01246S. Peter Sepitka sepitkap@math.muni.cz 1 Department of Mathematics and Statistics, Faculty of Science, Masaryk University, Kotlářská 2, CZ-61137 Brno, Czech Republic Springer 1140 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 Classical theory of system (H) involves a complete controllability (or equivalently an identical normality) assumption, see e.g. [1,5,8,11,14,16]. Recently in [19-23] the author and Simon Hilscher developed the theory of principal and antiprincipal solutions at infinity for a nonoscillatory and possibly abnormal system (H) satisfying the Legendre condition 5(0 > 0 for all? e [a, oo). (1.1) We showed the existence of principal solutions (X, U) at infinity with all ranks of X(t) in the range between n — doo and n, where doo is the maximal order of abnormality of (H) (defined in (2.6) below), and derived their classification and limit properties with antiprincipal solutions at infinity. These results motivated the investigations in [20,22], where we studied conjoined bases (X, U) of (H) with eventually the same image of X(t). According to [20, Definition 6.3] we say that such conjoined bases of (H) form a genus Q. We proved that every genus Q can be represented by an orthogonal projector Rg(t) satisfying the Riccati type matrix differential equation R£-A(t)Rg -RgAT{t) + Rg[A{t) + AT{t)]Rg = 0, te[a,oo). (1.2) This allowed to obtain under (1.1) a geometric description of the set T of all genera of conjoined bases of a nonoscillatory system (H), being a complete lattice [22, Theorem 4.8]. In this paper we extend the theory of genera of conjoined bases to arbitrary systems (H), for which the Legendre condition is not assumed and/or the system may be oscillatory. More precisely, in the new general definition of a genus corresponding to a conjoined basis (X, U) of (H) we consider the subspace ImX(0 + ImtfAoo(0, (1.3) where 7?Aoo(?) is the orthogonal projector onto the maximal subspace of the values at t of eventually degenerate solutions (x = 0, u) of (H). We refer to Sect. 2 for precise definitions of these notions, including the notions of a conjoined basis and the (non)oscillation of system (H). When the Legendre condition (1.1) holds and system (H) is nonoscillatory, then the subspace in (1.3) eventually coincides with the image of X(t), which yields the previous definition of a genus of conjoined bases in [20, Definition 6.3] for this special case. In the more general context of (1.3) we then derive (Theorems 4.7 and 4.8) a characterization of a genus Q of conjoined bases in terms of the orthogonal projector Rg (?) onto the subspace in (1.3), where Rg(t) satisfies the Riccati equation (1.2). This then leads to a natural ordering on the set T of all genera of conjoined bases (Theorem 4.12), as well as to a result stating that the set T forms a complete lattice (Theorem 4.14). This way we obtain direct generalizations of the results in [22] to the case of system (H) without the Legendre condition (1.1) or system (H) which is allowed to be oscillatory. The above results are based on the analysis (Theorems 3.9, 3.10, and 3.13) of the relationship between the orthogonal projectors solving the Riccati equation (1.2) and certain subspaces associated to the linear control system x' — A{t)x + B(t) u, te[a,oo). (1.4) In this analysis we make use of the properties of filters in ordered sets (lattices). The presented theory of genera of conjoined bases (in Sect. 4) is interesting by itself, as well as it is motivating for subsequent research. It is applied in our subsequent paper [18, Theorem 7.8 and Remark 7.9] in the study of distinguished solutions of Riccati matrix differential equations for possibly abnormal linear Hamiltonian systems (compare with the theory of Riccati matrix equations in [5,14,15]). Reference [18] appeared during the review process of this paper. Moreover, system (H) with the maximal order of abnormality doo — 0 Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1141 satisfies the condition of weak disconjugacy, see [7, Definition 2.2], [9, Remark 2.6, part 2], and [10, Lemma 5.5], so that the presented theory can be applied also to weakly disconjugate system (H). The paper is organized as follows. In Sect. 2 we recall basic notions from the theory of linear Hamiltonian systems. In Sect. 3 we study the properties of solutions of the linear control system (1.4) with the aid of the Riccati equation (1.2). Finally, in Sect. 4 we present the main results of this paper regarding the new concept of a genus of conjoined bases of (H). We also provide several examples illustrating this new theory. 2 Linear Hamiltonian Systems and Their Solutions In this section we review needed notions and results about linear Hamiltonian systems. Following a common convention, matrix solutions (X, U) of (H) will be denoted by the capital letters, where X, U : [a, oo) —> M"x" are piecewise continuously differentiable matrix-valued functions on [a, oo). A solution (X, U) of (H) is called a conjoined basis if rank (XT(t), UT(t))T — n and XT (t) U(?) is symmetric at some and hence at any ? e [a, oo). The principal solution (Xa, Ua) at the point a e [a, oo) is an example of such a conjoined basis. It is defined as the solution of (H) with the initial conditions Xa (a) — 0 and Ua(a) — I. The oscillation of conjoined bases of (H) satisfying (1.1) is defined via the concept of proper focal points, see [27, Definition 1.1] and [6,24,25]. However, this concept will not be explicitly needed in this paper. By [26, Definition 2.1], a conjoined basis (X, U) of (H) is called nonoscillatory if there exists a e [a, oo) such that Ker X(t) is constant on [a, oo). In the opposite case (X, U) is called oscillatory. The main result of [26] then describes the nonoscillatory behavior of conjoined bases of (H). Proposition 2.1 Assume that the Legendre condition (1.1) holds. Then there exists a nonoscillatory conjoined basis of (H) if and only if every conjoined basis of (H) is nonoscillatory. Based on this result we say that system (H) is {non)oscillatory if one and hence all conjoined bases of (H) are (non)oscillatory. In this paper we will use orthogonal projectors. If V is a subspace of W1, then we denote by Vy the orthogonal projector onto V. That is, Vy is a symmetric and idempotent n x n matrix such that Im Vy = V = Ker (/ - Vy) and Ker Vy = V± = Im (/ - Vy). Orthogonal projectors are easily constructed by using Moore-Penrose pseudoinverses. More precisely, for a matrix M e Wnxn we denote by el"" its Moore-Penrose pseudoinverse. Then the matrix MM^ is the orthogonal projector onto Im M and the matrix M^M is the orthogonal projector onto ImM^ — ImMr. Moreover, rankM — rank MM^ — rank M. For a general theory of pseudoinverse matrices we refer to [2], [3, Chapter 6], and [4, Section 1.4]. Given a conjoined basis (X, U) of (H), by its kernel, resp. image, we mean the kernel, resp. image, of X. Furthermore, we define on [a, oo) the orthogonal projectors onto the subspaces ImXr(f) andImX(?) by ^(0 == ?W(0 - X\t) X(t), R(t) := VimX(t) = X(t) X\t). (2.1) If (X, U) has constant kernel on [a, oo) c [a, oo), then P(t) is constant on [a, oo) and we set P ■- P(t) on [a, oo). (2.2) Springer 1142 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 In this case (X, U) has constant rank r on [a, oo) with r :— rank X(t) — rank P — rank R(t) on [a, oo). (2.3) The following result is from [19, Theorem 4.2], in which we observe that condition (1.1) is not needed. Proposition 2.2 Let (X, U) be a conjoined basis of (H) with constant kernel on [a, oo) and let P and R(t) be the corresponding matrices in (2.2) and (2.1). Then the equalities Im [U(t) (I - P)] = Ker R(t), B(t) = R(t) B(t) = B(t) R(t) (2.4) hold for all t e [a, oo). Moreover, the matrix R(t) solves the Riccati equation (1.2) on [a, oo). Following the standard notation used in [13, Section 3] and [22, Section 2], we denote by A[a, oo) the linear space of n-dimensional piecewise continuously differentiable vector-valued functions u which satisfy the equations u' — —AT(t) u and B(t) u — 0 on [a, oo). The functions u e A[a, oo) then correspond to the solutions (x = 0, u) of system (H) on [a, oo). The space A [a, oo) is finite-dimensional with d[a, oo) := dim A [a, oo) < n. The number d[a, oo) is called the order of abnormality of system (H) on the interval [a, oo). For a given a e [a, oo) we denote by At[a, oo) the subspace in W1 of values of functions u e A[a, oo) at the point t e [a, oo), i.e., At[a, oo) :— jcel", u(t) — c for some u e A[a, oo)}, te[a,oo). (2.5) It is easy to see that the subspace At[a, oo) is finite-dimensional with dimA([a, oo) = d[a, oo) for all t e [a, oo). We note that the set A[t, oo) is nondecreasing in t on [a, oo) with respect to the usual ordering in the set of linear subspaces of W1 and hence it is eventually constant. This means that the integer-valued function d[t, oo) is nondecreasing, piecewise constant, and right-continuous on [a, oo). In particular, there exists the limit :— lim d[t, oo) — max d[t, oo), 0 < doo < n, (2.6) t^oo te[a,oo) which we call the maximal order of abnormality of (H). Moreover, the definition of the quantity d^ in (2.6) and the monotonicity of the function d[t, oo) then imply the existence of the point e [a, oo) satisfying otoo — min{a e [a, oo), d[a, oo) — doo}. (2-7) From (2.6) and (2.7) we then obtain that the subspace Ala^, oo) satisfies the equalities A[aoo, oo) — lim A[a, oo) — max A[a, oo), (2.8) a^oo ae[a,oo) A[a, oo) = A[aoo, oo), a e [otoo, oo)- (2-9) 3 Auxiliary Results on Linear Control Systems In this section we present some auxiliary results about the solutions of the first order linear differential equation (1.4), where A, B : [a, oo) —> Wixn are given piecewise continuous matrix-valued functions. Our main results describe the relationship between the subspaces of values of the solutions of system (1.4) and the orthogonal projectors solving the Riccati Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1143 equation (1.2), see Theorems 3.9 and 3.13. These properties will be utilized in Sect. 4 in order to develop correctly the concept of a genus of conjoined bases of system (H). Solutions of (1.4) are considered to be pairs of n-dimensional vector-valued functions (x,u) such that u(t) is piecewise continuous on [a, oo) and x(t) is piecewise continuously differentiable on [a, oo). In the literature such a pair (x, w) is sometimes termed as admissible, while (1.4) is then called the equation of motion. This terminology is motivated by the variational analysis and control theory, where the solutions of (1.4) are related with a certain type of quadratic functionals, see e.g. [5,11,12,14,17]. By the symbol S we will denote the linear space of the first components of solutions of (1.4), i.e., S :— {x : [a, oo) —> M.n, (x, u) is admissible for somew}. Let t € [a, oo). For a given subspace V c S we will denote by Vt the subspace of W consisting of the values of functions i e Vat the point t, that is Vt := |cel", x(t) = c for some x e V}. (3.1) The main results of this section are formulated in terms of orthogonal projectors Z(t) which solve the symmetric Riccati matrix differential equation (1.2), i.e., Z'- A(t)Z-ZAT(t) + Z[A(t) + AT(t)]Z = 0, t e [a, oo). (3.2) We recall from Sect. 2 that the matrix M eWlxn is an orthogonal projector if M is symmetric and idempotent, i.e., the equalities M — MT and M2 — M hold. In the following three propositions and one remark we collect basic results about the solutions Z(t) of (3.2) which are orthogonal projectors for all t e [a, oo), see [22, Section 3]. Proposition 3.1 LetZo e M"x" be an orthogonal projector and let to e [a, oo) be fixed. Then the unique solution Z(t) of (3.2) which satisfies the initial condition Z(?rj) — Zrj exists on the whole interval [a, oo) and the matrix Z(t) is an orthogonal projector for all t e [a, oo). Proposition 3.2 Let Z\{t) and Z^if) be two orthogonal projectors which solve (3.2) on [a, oo). Then the inclusion Im Z\ (t) C Im Z2(t) holds for every t e [a, oo) if and only if the inclusion Im Zi(?rj) 5= ImZ2(?o) holds for some to e [a, oo). Proposition 3.3 Let Z\{t) and Z2(t) be two orthogonal projectors which solve (3.2) on [a, oo). For each t e [a, oo) denote by Z(t) and Z(t) the orthogonal projectors onto the subspaces ImZi(?) D \v&Ziif) and ImZi(?) + \v&Ziit), respectively. Then the matrices Z(t) and Z(t) satisfy (3.2) on [a, oo). Remark 3.4 The matrix Z(t) is a solution of (3.2) if and only if / — Z(t) is a solution of the Riccati equation Y'+ AT(t)Y+ Y A(t)-Y[A(t) + AT(t)]Y = 0, te[a,oo), (3.3) which is obtained from (3.2) by changing the coefficient A(t) to — AT(t). Throughout this section we will use the following notation. By the symbol C(A) we will denote the set of all orthogonal projectors Z(t) which solve (3.2) on the whole interval [a, oo). Furthermore, the symbol C(A, B) will denote the set of all orthogonal projectors Z(t) which solve (3.2) on [a, oo) and in addition satisfy the inclusion Im B(t) c Im Z(t) for all t e [a, oo). (3.4) Springer 1144 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 Remark3.5 (i) The results in Propositions 3.1-3.3 show that the set C(A) can be ordered with respect to inclusion of subspaces Im Z(t). More precisely, L(A) is a complete lattice with the least element Z(t) = 0 on [a, oo) and the greatest element Z(t) = I on [a, oo). In particular, the matrices Z(t) and Z(t) from Proposition 3.3 represent the infimum and supremum of Z\(t) and Zi(t), respectively, in this ordering. We also note that the lattice C(A) is isomorphic to the complete lattice of all subspaces in W1. (ii) The observation in Remark 3.4 implies that the complete lattice L(—AT) associated to equation (3.3) is isomorphic to the dual lattice to L(A). In particular, the orthogonal projector Z(t) belongs to L(A) if and only if the orthogonal projector / — Z(t) belongs to L(—AT). From Remark 3.5(i) and the obvious relation L(A, 5) c L(A) we obtain immediately that the matrices Z(t) belonging to set L(A, 5) can be also ordered with respect to inclusion of the subspaces Im Z(t). Moreover, in the next proposition we prove that the ordered set C(A, B) is a sublattice of C(A). Proposition 3.6 The ordered set C(A, 5) is a sublattice of the complete lattice C(A). Proof Let Z\(t) and Zi(t) be two orthogonal projectors belonging to L(A, B). Moreover, let Z(t) and Z(t) be the orthogonal projectors in Proposition 3.3 and Remark 3.5(i) which respectively correspond to the infimum and supremum of Z\(t) and Zi(t) in the lattice C(A). We will show that the matrices Z(t) and Z(t) belong to the set L(A, B). Indeed, the two projectors Z(t) and Z(t) solve (3.2) on the whole interval [a, oo), by Proposition 3.3. Moreover, the inclusions lmB(t) c Im Z\ it) and Im Bit) c Im Zi(t) on [a, oo) imply that ImS(0 c ImZi(0 nimZ2(0 = lmZ(t), lmB(t) c ImZi(?) +ImZ2(0 = ImZ(r) for every t e [a, oo). This shows that the matrices Z(t) and Z(t) satisfy condition (3.4) and hence, they belong to C(A, B). Therefore, the ordered set C(A, B) is a sublattice of the lattice £(A) and the proof is complete. □ In the following lemma we derive an auxiliary property of solutions of (1.4). Lemma 3.7 Let (x, u) be an admissible pair and let Z(t) be an orthogonal projector belonging to C(A). Then the vector w(t) := [I — Z(t)] x(t) satisfies on [a, oo) the equation w' - {A(t) - Z(t) [A(0 + AT(t)]} w-[I - Z(t)] 5(0 ii (0 = 0. (3.5) Proof Let (x, u) and Z(t) be as in the statement. By using (1.4) and (3.2) we have on [a, oo) w' = -Z'x + (/ - Z) x' = [-AZ - ZAT + Z(A + AT)Z]x + (I- Z) (Ax + Bu) = {A-Z[A + AT]}w + (I - Z) Bu. Thus, the vector w(t) satisfies the linear differential equation (3.5) on [a, oo). □ Remark 3.8 Let to e [a, oo) be fixed. Using the variation of constant formula applied to (3.5) we obtain for the function w(t) in Lemma 3.7 the expression w(t) = $(?, t0) w(t0) + $(?, t0) f $>~l(s, t0) [I - Z(s)] B(s) u(s)ds, te[a,oo), J t0 (3.6) where to) is the fundamental matrix of the equation y' — {A(t) — Z(t) [A(t) + AT (t)]} y for t e [a, oo) with <£>(to, to) = I. Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1145 In the first main result of this section we provide a fundamental connection between the structure of the lattice C(A, B) and the subspaces Vt in (3.1). For this purpose we now recall the standard terminology for lattices (M., <)■ Namely, a subset T c is called a filter of the lattice A4 if T is a sublattice of A4 such that for any / e T and i e M the condition / < x implies x e T. Moreover, for a given f e A4 the subset F :— {x e M., / ^ x} is called the principal filter of A4 generated by f. In particular, we describe all principal filters of £(A, B) in terms of the subspaces Vt. Theorem 3.9 Let Z(t) be an orthogonal projector belonging to the lattice C(A, B) and let Z be the principal filter ofC(A, B) generated by Z(t). Then the following statements hold. (i) For any given subspace V 5= (Zoo the matrix RAoo(t) defined in (4.1) is the orthogonal projector onto the set (A, [a, oo)) ^ on [a, oo), i.e., RA00(t)=Vu±, where U, := A,[a, oo), t e [a, oo). (4.2) (ii) We note that in agreement with Remark 3.15 the orthogonal projector RAoo(t) is the least element of the complete lattice C(A, B)coin the following auxiliary result we derive a relation between conjoined bases of (H) and orthogonal projectors of the set C(A, B)^. Lemma 4.2 Let (X, U) be a conjoined basis of (H). For each t e [aco, oo) denote by Z(t) the orthogonal projector onto the subspace Im X(t) + Im RAoo(t). Then the matrix Z(t) belongs to the set C(A, £)co- Proof Let (X, U) and Z(t) be as in the statement. For a given c e W1 consider the vector solution (x, u) :— (Xc, U c) of (H). Since the pair (x, u) solves the equation of motion (1.4), it is an admissible pair. Moreover, let V be the subspace of the first components of all such admissible pairs (x, w), that is, the subspace of all vector-valued functions x of the form x — Xc for some c e M". In particular, the equality Vt — Im X(t) holds for all t e [a, oo). From Remark 4.1(h) and Theorem 3.10(i) it then follows that the orthogonal projector Z(t) belongs to C{A, B)^. □ In [20, Definition 6.3] we introduced a genus of conjoined bases for a nonoscillatory system (H). Below we extend this notion to general possibly oscillatory system (H). We can observe that the orthogonal projector RAoo(t) defined in (4.1) plays a crucial role in this extension, see also the comments in Remark 4.17(i). Also, below we do not require the validity of the Legendre condition (1.1). Definition 4.3 (Genus ofconjoined bases) Let (Xi, Lq) and(X2, U2) be two conjoined bases of (H). We say that (Xi ,Ui) and (X2, U2) have the same genus (or they belong to the same genus) if there exists a e [a^, 00) such that ImXi(0 + lmRAoo(t) — ImX2(?) + ImRAoo(t), t e [a, 00), where is defined in (2.7). Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1149 Remark 4.4 From Definition 4.3 it follows that the relation "having (or belonging to) the same genus" is an equivalence relation on the set of all conjoined bases of (H). Therefore, there exists a partition of this set into disjoint classes of conjoined bases of (H) with the same genus. This allows to interpret each equivalence class Q as a genus itself. In the following result we provide a fundamental property of conjoined bases of (H) which belong to the same genus. Theorem 4.5 Let (Xi, U\) and (X2, U2) be conjoined bases of (H). Then the following statements are equivalent. (i) The conjoined bases (X\, U\) and (X2, U2) belong to the same genus. (ii) The equality ImXi(t) + lmRAoo(t) — ImA^CO + lmRAoo(t) holds for every t e [otoo, 00). (iii) The equality Imli(f) + lmRAoo(t) — lmX2(t) + lmRAoo(t) holds for some t e [Otoo, oo). Proof Let (Xi,U\) and (X2, U2) be as in the theorem and let Z1 (?) and Z2(t) be the orthogonal projectors in Lemma 4.2 associated to the functions X\{t) and on [aco, 00). In particular, the matrices Z\(t) and Z2OO solve the Riccati equation (3.2) on {a^, 00). If (Xi, Ui) and (X2, U2) belong to the same genus, then the equality Z\(t) — Z2OO holds on [a, 00) for some a > a^, by Definition 4.3 and by the uniqueness of orthogonal projectors. From Proposition 3.1 it then follows that Z\(t) — Z2OO for all t e [otoo, °o), showing (ii). Conversely, (ii) implies (i) trivially. Finally, by using Proposition 3.1 once more and the uniqueness of solutions of (3.2) we obtain the equivalence of statements (ii) and (iii), which completes the proof. □ Let Q be a genus of conjoined bases of (H) and let (X, U) be a conjoined basis of (H) belonging to Q. The definition of genus and the results in Theorem 4.5 imply that for all t e [aco, 00) the subspace Im X(t) + Im RAoo(t) does not depend on the particular choice of such a conjoined basis (X, U). Therefore, the orthogonal projector onto Im X(t)+Im RAoo(t), i.e., the matrix Rg(t) := Vv„ where Vt := ImX(t) + ImRAoo(t), t e [««,, 00), (4.3) is uniquely determined for each genus Q. Remark 4.6 We note that the definition of the matrix Rg(t) in (4.3) implies that for every conjoined basis (X, U) of (H) in the genus Q the associated orthogonal projector Z(t) in Lemma 4.2 satisfies Z(t) — Rg(t) on [otoo, 00) • In particular, this shows that the orthogonal projectors Rg(t) are elements of the complete lattice C(A, B)^. The next two theorems provide a classification of all genera Q of conjoined bases of (H) in terms of their associated orthogonal projectors Rg(t) in (4.3). More precisely, we show that the matrices Rg(t) solve the Riccati equation (1.2) on [otoo, 00) and satisfy the inclusion Im RAoo(t) c lmRg(t) for all t e [otoo, 00)• Theorem 4.7 Let Q be a genus of conjoined bases of (H) and let Rg(t) be the orthogonal projector defined in (4.3). Then the matrix Rg(t) is a solution of the Riccati equation (1.2) on [ffico, 00) and the inclusion lmRAoo(t) C lmRg(t) holds for every t e [oioo, °°)- Springer 1150 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 Proof From Remark 4.6 we know that the matrix Rg(t) belongs to the complete lattice £(A, B)co- Therefore, the function Rg(t) solves (1.2) on [otoo, oo). Moreover, by Remark 4.1(h) the matrix RAoo(t) defined in (4.1) is the least element of C(A, B)^, which in turn yields the condition Im RAoo (?) c Im Rg (?) for all ? e [a^, oo). □ Theorem 4.8 Let f3 e [otoo, oo) be fixed and let R e Wixn be an orthogonal projector satisfying Im RAoo (/J) C Im R. Then there exists a unique genus Q of conjoined bases of (H) such that its corresponding orthogonal projector Rg(t) in (4.3) satisfies Rg(fi) — R. Proof Let R and f3 be as in the theorem and consider the solution (X, U) of (H) given by the initial conditions X (/?) = R - RAoo (/?) and U (/?) = I - R + 7?Aoo (/?). First we will prove that (X, £/) is a conjoined basis. By using the identities R RAoo(/3) — RAoo(/3) — RAoo(P) R we obtain that the matrix X7'(/?) U(fi) = [R - RAoo(P)] [I - R + RAoo(P)] = 0 is symmetric. Moreover, the equalities rank [XT(fi), UT (ft)] = rank [/? — ieAoo(/3), 7-# + #AooG6)] =« hold, because Ker [R - RAoo(fi)] n Ker [7 - 7? + RAoo(P)] = {0}. This shows that (X, U) is a conjoined basis of (H). Let Q be the genus of conjoined bases of (H) such that (X, U) e Q and let Rg (?) be its corresponding matrix defined in (4.3). The equality Im [R — RAoo (/J)] = ImR n KerRAoo(P) implies that Im [R - RAoo(P)] n Im#AooG6) = {0}. Therefore, the subspace Vt in (4.3) satisfies Vp =\m[R - Raoo{p)]®\mRaoo{p) = (im R n Ker tfAoo(/3)) © Im /?Aoo(/3) - Im R. On the other hand, the identity Vp — Im Rg (/J) holds. Thus, we have that Im Rg (/J) — Im R, which by the uniqueness of the orthogonal projectors yields the equality Rg (/J) — R. Finally, since the matrix Rg (?) solves the Riccati equation (1.2) or (3.2) on [a^, oo) by Theorem 4.7 and since the solutions of (3.2) are unique by Proposition 3.1, it follows that the genus Q is uniquely determined by Rg (?), and hence by the orthogonal projector R. □ Remark4.9 Combining the results from Remark 4.1(h) and Theorems 4.7 and 4.8 allows to strengthen the observation in Remark 4.6. Namely, the matrices Rg(t) in (4.3) associated to genera Q of conjoined bases of (H) coincide with the elements of the complete lattice £(A, B)oo. We also note that every such orthogonal projector Rg(t), as a solution of (1.2) on [aco, oo), has constant rank rg on the whole interval [a^, oo). In this context, we may adopt for the number rg the terminology rank of the genus Q and write rank Q := rQ, compare also with [21, Remark 6.4]. In particular, n — dco — rank Q < n holds. Based on the results in Theorems 4.7 and 4.8 we can now introduce an ordering on the set of all genera of conjoined bases of (H) which corresponds to the ordering in the lattice Definition 4.10 Let Q and TL be two genera of conjoined bases of (H) and let Rg(t) and R-u(t) be their corresponding orthogonal projectors in (4.3), respectively. We say that the genus Q is below the genus TL (or that the genus TL is above the genus Q) and we write Q < TL if the inclusion Im Rg(t) c Im R-n(t) holds for all ? e [otoo, oo). Remark 4.11 We note that the genera Q and TL satisfy Q < TL if and only if the inclusion ImRg(f3) c lmRn(/3) holds for some f3 e [otoo, oo). This follows from Definition 4.10 and Proposition 3.2, because the matrices Rg(t) and R-n(t) are solutions of the Riccati equation (1.2) on [aco, oo), by Theorem 4.7. By the symbol T we will denote the set of all genera of conjoined bases of (H). Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1151 Theorem 4.12 The relation the principal solution (Xa, Ua) at the point a belongs to Gmin. Proof Let Q be the genus of conjoined bases of (H) such that (X, U) € Q and let Rg (?) be its representing orthogonal projector in (4.3). In particular, the inclusions Im X(t) c Im Rg(t) and lmRAoo(t) c lmRg(t) hold on [aco, oo). If Q is equal to the minimal genus Qmin, then Rg(t) — RAoo(t) for all ? e [otoo, oo), by Remark 4.13(h), and in turn we obtain that lmX(t) c lmRg(t) — lmRAoo(t) for every ? e [otoo, oo). Conversely, let f3 e [otoo, oo) be fixed and suppose that the inclusion lmX(/3) c lmRAoo(/3) holds. The matrix Rg(/3) then satisfies the equalities Im Rg(/3) — lmX(/3) + lm RAoo(/3) — Im RAoo(/3), which by the uniqueness of orthogonal projectors means that Rg(/3) — RAoo(P)- Therefore, the genus Q is equal to Gmin, by Theorem 4.8 and Remark 4.13(h). Finally, if (Xa, Ua) is the principal solution of (H) at some point a e [otoo, oo), then Xa(a) — 0 and hence, the inclusion lmXa(a) c lmRAoo(a) holds. This shows by the first part that (Xa, Ua) belongs to the minimal genus Gmin- □ In the next result we derive important properties of nonoscillatory conjoined bases from a given genus G- Springer 1152 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 Proposition 4.16 Let Q be a genus of conjoined basis of (H) with the corresponding orthogonal projector Rg (?) in (4.3). Moreover, let (X, U) be a conjoined basis of (H) with constant kernel on [a, oo) C [a^, oo) such that (X, U) belongs to Q and let R(t) be the matrix defined in (2.1). Then the equality Rg(t) — R(t) holds for all t e [a, oo). Proof Let P be the constant orthogonal projector in (2.2) associated to (X, U) and put (X, U) := (X(I-P), [/(/-P)). By using the identity X(t) P = X(t)ior every? e [a, oo) we obtain that for any c e W1 the pair (Xc, Uc) is a vector solution of (H) satisfying X(t) c = 0 on [a, oo). Therefore, the function Uc e A[a, oo). In particular, this means that the inclusion Im [U(?) (/ — P)] c At[a, oo) holds for all t e [a, oo). Let RAoo(t) be the orthogonal projector in (4.1) associated to the subspace A, [otoo, oo) on [otoo, oo). By using the first equality in ( ) and formula (4.2) in Remark 4. l(i) we have that Ker R(t) = Im [1/(0 (/ - P)] c At[a, oo) - [ImPAoo(0 ]x - Ker RAoo(t) (4.4) for every t e [a, oo). In fact, taking the orthogonal complements implies that the relation in (4.4) is equivalent with the inclusion Im RAoo(t) ^ Im R(t) on [a, oo). Moreover, according to (4.3) the matrix Rg(t) satisfies lmRg(t) — lmR(t) + ImPAoo(?) — ImP(0 on [a, oo), which by the uniqueness of orthogonal projectors gives the equality Rg(t) — R(t) for all t e [a, oo). □ Remark 4.17 Assume that the Legendre condition (1.1) holds and that system (H) is nonoscil-latory. By Proposition 2.1 this means that every conjoined basis of (H) is nonoscillatory. In particular, for any conjoined basis (X, U) of (H) there exists a e [otoo, oo) such that (X, U) has constant kernel on [a, oo). Moreover, let Q be the genus of conjoined bases such that (X, U) € Q and let Rg(t) be its corresponding orthogonal projector in (4.3). Then the rank rQ of Q defined in Remark 4.9 coincides with the rank r of any conjoined basis (X, U) of the genus Q defined in (2.3). By Proposition 4.16 we have that Im X(t) — Im Rg(t) on [a, oo). Therefore, from (4.3) and Definition 4.3 it follows that two conjoined bases (X\, U\) and (X2, U2) of (H) belong to the same genus if and only if there exists a e [otoo, 00) such that the equality ImXi(?) — ImX2(0 hold for every t e [a, 00). This observation shows that the concept given in Definition 4.3 generalizes the definition of genus of conjoined bases introduced in [20, Definition 6.3] for nonoscillatory system (H). We also note that the result about the structure of the set of all genera of conjoined bases presented in this section are in full agreement with the corresponding results in [22, Section 4]. In the following examples we illustrate the new theory of genera of conjoined bases of (H). We refer to [22, Section 5] for examples of nonoscillatory systems (H) with (1.1). Thus, in the first example we consider an oscillatory system (H) satisfying the Legendre condition Example4.18 Let n = 2 and A(t) = 0, 5(0 = diag{l, 0}, and C(t) = diag{-l, 0} on [0, 00). The principal solution (Xa, Ua) at the point a — 0 has the form Since 5(0 > 0 on [0, 00) and the matrix Xa(t) in (4.5) changes its kernel at each t — kir, k e N, system (H) is oscillatory, by Proposition 2.1. Moreover, we have doo — 1, otoo — 0, and PAoo(0 = diag{l, 0} on [0, 00). Therefore, there exist only two genera of conjoined bases, i.e., the minimal genus Qmin with the corresponding orthogonal projector Rgmin(t) — (1.1). (4.5) Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1153 ^aoo(0 on [0, oo) and the maximal genus Qmax with the corresponding orthogonal projector RgmBX(t) = I on [0, oo). In particular, by (4.5) we have the identity ImXa(?) + ImRAoo(?) = ImRAoo(?) = ImRgmia(?), ? G [0, oo), and hence, the conjoined basis (Xa, Ua) belongs to the minimal genus Qmin > we also claim in Theorem 4.15. On the other hand, the conjoined basis (X, U) of the form is an element of the maximal genus CJmax> because by (4.6) we have lmX(t) +lmRAoo(t) — R2 — lmRGmm(t), t e [0, oo). In the second example we consider a system (H) which does not satisfy the Legendre condition (1.1). We note that this system is neither oscillatory nor nonoscillatory in the sense of Proposition 2.1. Example 4.19 For n — 3 and a — Owe consider system (H) with A(?) — C(?) = 0 and /0 0 0 \ 5(0 = 0 0 e' on [0, oo). (4.7) From (4.7) we can easily see that the matrix B(t) is indefinite for all t e [0, oo) and hence, the Legendre condition (1.1) does not hold. Moreover, we have doo — 1, otoo — 0, and RAoo(t) = diag {0, 1, 1} on [0, oo). Therefore, system (H) is abnormal and it possesses only two genera of conjoined bases. Namely, there is the minimal genus Q — Qram represented by the orthogonal projector Rg(t) — RAoo(t) on [0, oo), which contains for example the conjoined bases {Xi(t),Ui(t)) = (X2(t), U2(t)) =^0 1 e'j,/j, t G [0, oo), and there is the maximal genus Q — ^7max represented by the orthogonal projector Rg (?) = / on [0, oo), which contains for example the conjoined bases (X3(t), U3(t)) - (/, 0), (X4(t), I/4(0) = 01 e' , / , ? G [0, oo). We note that the conjoined bases (X\, Ui) and (X3, U3) have constant kernel on [0, oo) with the corresponding constant projectors Pi — diag{0, 1,1} and P2 — I in (2.2). On the other hand, the conjoined bases (X2, U2) and (X4, U4) do not have constant kernel on any nondegenerate subinterval in [0, 00) and their associated orthogonal projectors P2(t) and P4(?) in (2.1) are 1 /0 0 0 \ j /e2' + 1 0 0 \ e +1 W e2'/ \ 0 e' e2'/ Spring 1154 Journal of Dynamics and Differential Equations (2020) 32:1139-1155 respectively. In particular, by using the terminology of Sect. 2 the conjoined bases (Xi, U\) and (X3, U3) are both nonoscillatory on [0, 00), while the conjoined bases (X2, U2) and (X4, U4) are both oscillatory on [0, 00). This observation shows that in the absence of the Legendre condition (1.1) the statement of Proposition 2.1 does not hold, i.e., in this case it is not possible to classify system (H) as nonoscillatory or oscillatory. In spite of that both conjoined bases (X2, U2) and (X4, U4) have constant rank on the interval [0, 00). Namely, in agreement with (2.3) we have Y2 — rank = 1 and Y4 — rank X^(t) = 2 on [0, 00). Acknowledgements The author is grateful to Professor Roman Simon Hilscher for consultations regarding the subject of this paper. The author wish to thank an anonymous referee for detailed reading of the paper and valuable suggestions which improved the presentation of the results. References 1. Atkinson, F.V.: Discrete and Continuous Boundary Problems. Academic Press, New York (1964) 2. Ben-Israel, A., Greville, T.N.E.: Generalized Inverses: Theory and Applications, 2nd edn. Springer, New York (2003) 3. Bernstein, D.S.: Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory. Princeton University Press, Princeton (2005) 4. Campbell, S. L., Meyer, CD.: Generalized Inverses of Linear Transformations, Reprint of the 1991 corrected reprint of the 1979 original, Classics in Applied Mathematics, vol. 56. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (2009) 5. Coppel, W.A.: Disconjugacy Lecture Notes in Mathematics, vol. 220. Springer, Berlin (1971) 6. Elyseeva, J.: Comparison theorems for conjoined bases of linear Hamiltonian differential systems and the comparative index. J. Math. Anal. Appl. 444(2), 1260-1273 (2016) 7. Fabbri, R., Johnson, R., Novo, S., Núňez, C: Some remarks concerning weakly disconjugate linear Hamiltonian systems. J. Math. Anal. Appl. 380(2), 853-864 (2011) 8. Hartman, P.: Ordinary Differential Equations. Wiley, New York (1964) 9. Johnson, R., Novo, S., Núňez, C, Obaya, R.: Uniform weak disconjugacy and principal solutions for linear Hamiltonian systems, In: Recent Advances in Delay Differential and Difference Equations (Balatonfuered, Hungary, 2013), Springer Proceedings in Mathematics & Statistics, vol. 94, pp. 131-159. Springer, Berlin (2014) 10. Johnson, R., Obaya, R., Novo, S., Núňez, C, Fabbri, R.: Nonautonomous Linear Hamiltonian Systems: Oscillation, Spectral Theory and Control Developments in Mathematics, vol. 36. Springer, Cham (2016) 11. Kratz, W.: Quadratic Functionals in Variational Analysis and Control Theory. Akademie Verlag, Berlin (1995) 12. Kratz, W.: Deflniteness of quadratic functionals. Analysis 23(2), 163-183 (2003) 13. Reid, W.T.: Principal solutions of nonoscillatory linear differential systems. J. Math. Anal. Appl. 9, 397^123 (1964) 14. Reid, W.T.: Ordinary Differential Equations. Wiley, New York (1971) 15. Reid, W.T.: Riccati Differential Equations. Academic Press, New York (1972) 16. Reid, W.T.: Sturmian Theory for Ordinary Differential Equations. Springer, New York (1980) 17. Speyer, J.L., Jacobson, D.H.: Primer on Optimal Control Theory Advances in Design and Control, vol. 20. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (2010) 18. Sepitka, P.: Riccati equations for linear Hamiltonian systems without controllability condition. Discrete Contin. Dyn. Syst. 39(4), 1685-1730 (2019) 19. Sepitka, P., Simon Hilscher, R.: Minimal principal solution at infinity for nonoscillatory linear Hamiltonian systems. J. Dyn. Differ. Equ. 26(1), 57-91 (2014) 20. Sepitka, P., Simon Hilscher, R.: Principal solutions at infinity of given ranks for nonoscillatory linear Hamiltonian systems. J. Dyn. Differ. Equ. 27(1), 137-175 (2015) 21. Sepitka, P., Simon Hilscher, R.: Principal and antiprincipal solutions at infinity of linear Hamiltonian systems. J. Differ. Equ. 259(9), 4651^1682 (2015) 22. Sepitka, P., Simon Hilscher, R.: Genera of conjoined bases of linear Hamiltonian systems and limit characterization of principal solutions at infinity. J. Differ. Equ. 260(8), 6581-6603 (2016) 23. Sepitka, P., Simon Hilscher, R.: Reid's construction of minimal principal solution at infinity for linear Hamiltonian systems. In: Pinelas, S., Došlá, Z., Došlý, O., Kloeden, PE. (eds.) Differential and Difference Springer Journal of Dynamics and Differential Equations (2020) 32:1139-1155 1155 Equations with Applications, Proceedings of the International Conference on Differential & Difference Equations and Applications (Amadora, 2015), Springer Proceedings in Mathematics & Statistics, vol. 164, pp. 359-369, Springer, Berlin (2016) 24. Sepitka, P., Simon Hilscher, R.: Comparative index and Sturmian theory for linear Hamiltonian systems. J. Differ. Equ. 262(2), 914-944 (2017) 25. Simon Hilscher, R.: Sturmian theory for linear Hamiltonian systems without controllability. Math. Nachr. 284(7), 831-843 (2011) 26. Simon Hilscher, R.: On general Sturmian theory for abnormal linear Hamiltonian systems. In: Feng, W., Feng, Z., Grasselli, M., Ibragimov, A., Lu, X., Siegmund, S., Voigt, J. (eds.) Dynamical Systems, Differential Equations and Applications, Proceedings of the 8th AIMS Conference on Dynamical Systems, Differential Equations and Applications (Dresden, 2010), Discrete and Continuous Dynamical Systems, Supplement vol. 2011, pp. 684-691. American Institute of Mathematical Sciences (AIMS), Springfield (2011) 27. Wahrheit, M.: Eigenvalue problems and oscillation of linear Hamiltonian systems. Int. J. Differ. Equ. 2(2), 221-244 (2007) Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Springer APPENDIX B Paper by Sepitka (DCDS 2019) This paper entitled " Riccati equations for linear Hamiltonian systems without controllability condition" appeared in the journal Discrete and Continuous Dynamical Systems, 39 (2019), no. 4, 1685-1730, see item [' T] in the bibliography. This paper is dedicated to the memory of Professor Russell A. Johnson. 61 DISCRETE AND CONTINUOUS DYNAMICAL SYSTEMS Volume 39, Number 4, April 2019 doi:10.3934/dcds.2019074 pp. 1685-1730 RICCATI EQUATIONS FOR LINEAR HAMILTONIAN SYSTEMS WITHOUT CONTROLLABILITY CONDITION Peter Sepitka Department of Mathematics and Statistics, Faculty of Science, Masaryk University Kotlafska 2, CZ-61137 Brno, Czech Republic Dedicated to the memory of Professor Russell A. Johnson. (Communicated by Alberto Bressan) abstract. In this paper we develop new theory of Riccati matrix differential equations for linear Hamiltonian systems, which do not require any controllability assumption. When the system is nonoscillatory, it is known from our previous work that conjoined bases of the system with eventually the same image form a special structure called a genus. We show that for every such a genus there is an associated Riccati equation. We study the properties of symmetric solutions of these Riccati equations and their connection with conjoined bases of the system. For a given genus, we pay a special attention to distinguished solutions at infinity of the associated Riccati equation and their relationship with the principal solutions at infinity of the system in the considered genus. We show the uniqueness of the distinguished solution at infinity of the Riccati equation corresponding to the minimal genus. This study essentially extends and completes the work of W. T. Reid (1964, 1972), W. A. Coppel (1971), P. Hartman (1964), W. Kratz (1995), and other authors who considered the Riccati equation and its distinguished solution at infinity for invertible conjoined bases, i.e., for the maximal genus in our setting. 1. Introduction. Riccati differential equations for self-adjoint linear differential systems play fundamental role in mathematical research as well as in applications. Specifically, if n € N is a given dimension and A, B,C : [a, oo) —> R™x™ are given piecewise continuous matrix-valued functions such that B(t) and C(t) are symmetric, the Riccati matrix differential equation Q' + QA(t) +AT(t)Q + QB{t)Q - C(t) = 0 (R) is associated with the linear Hamiltonian system x' = A(t)x + B(t)u, v! = C(t)x - ATit)u, (H) see [7, 9, 17, 22, 23, 24]. It is known that under the Legendre condition B(t)>0 for alii € [a, oo) (1.1) the Riccati equation (R) has many applications in various disciplines, for example in the oscillation and spectral theory [2, 7, 17, 22, 23, 24], filtering and prediction theory [16, 23], calculus of variations and optimal control theory [1, 3, 8, 12, 10, 2010 Mathematics Subject Classification. 34C10. Key words and phrases. Linear Hamiltonian system, Riccati differential equation, genus of conjoined bases, distinguished solution at infinity, principal solution at infinity, controllability. This research was supported by the Czech Science Foundation under grant GA16-00611S. 1685 1686 PETER ŠEPITKA 19, 34, 35, 37, 38, 39], systems theory and control [14, 15], and others (engineering, etc.). In [20], Reid showed that when system (H) is completely controllable and nonosc-illatory, the Riccati equation (R) has the so-called distinguished solution Q(t) at infinity, which is the smallest symmetric solution of (R) existing on an interval [a, oo) for some a > a. In the subsequent paper [21], Reid derived the minimality of the distinguished solution of (R) at infinity also for a noncontrollable system (H) by considering invertible principal solutions (X, U) of (H) at infinity. Recently, the author and Simon Hilscher developed the theory of principal solutions at infinity for a general nonoscillatory and possibly abnormal system (H). We showed in [28, 29] the existence of principal solutions (X, U) at infinity with all ranks of X(t) in a specific range depending on the maximal order of abnormality rfoo of (H), their classification and limit properties with antiprincipal solutions at infinity [30], and the geometric structure of the set of all conjoined bases [31]. In particular, conjoined bases of (H) with eventually the same image of the first component form a genus Q, which can be represented by an orthogonal projector Rg(t) satisfying the Riccati type matrix differential equation Rg - A(t) Rg - Rg AT(t) + Rg [A(t) + AT(t)} Rg = 0. (1.2) This leads under (1.1) to a complete description of the set Y of all genera of conjoined bases of a nonoscillatory system (H), being a complete lattice [31, Theorem 4.8]. This result was recently extended to a possibly oscillatory system (H) in [27, Theorem 4.14]. In this paper we continue in the above study of linear Hamiltonian system (H) by developing the corresponding theory of Riccati matrix differential equations. The presented approach and results are novel in three directions: (i) we do not require any controllability assumption on system (H), (ii) for every genus Q we associate a Riccati equation Q' + Q A(t) +AT(t)Q + Q B{t) Q-C(t) = 0, (ft) where the coefficients Ait), Bit), and C(t) are given by A(t):=A(t)R0(t)-AT(t)[I-R0(t)}, 1 Bit) := Bit), C(t):=Re(t)C(t)Re(t), j ['' (iii) we show that every such a Riccati equation (ft) possesses a distinguished solution at infinity (defined in a suitable way), which corresponds to a principal solution of (H) at infinity from the genus Q. More precisely, given a genus Q of conjoined bases of (H), we show (Theorems 4.18 and 4.21) a fundamental connection between the symmetric solutions Q(t) of (TV) on [a, oo) with some a > a satisfying ImQ(t) C lmRg(t), te[a,oo), (1.4) and the conjoined bases (X, U) of (H) with constant kernel on [a, oo), which belong to Q. We define (Definition 7.1) a distinguished solution Q(t) at infinity for each Riccati equation (TV), which corresponds to a principal solution (X,U) of (H) at infinity in the genus Q. We also prove (Theorem 7.16) that for every symmetric solution Q(t) of (TV) on [a, oo) with (1.4) there exists a distinguished solution Q(t) riccati equations for linear hamiltonian systems 1687 of (TV) satisfying the inequality Q(t)>Q(t) on[a,oo). (1.5) The above results are particularly important for the minimal genus Q = Qm-ln, which is formed by the conjoined bases (X, U) of (H) with minimal possible rank of the matrix X(t), i.e., with rankX(t) = n — on [a, oo). In this case the associated distinguished solution Qm-ln(t) at infinity is unique and minimal among all symmetric solutions Q(t) of (7?.) satisfying (1.4). This latter situation generalizes the classical controllable results of Reid and Coppel [7, 20, 22], since in this case rfoo = 0 and the orthogonal projector Rg(t) = I on [a, oo), so that the Riccati equation (7?.) reduces to (R). We note that the original results by Reid [21, 23] for noncontrollable system (H) and Riccati equation (R) correspond in our new theory to the maximal genus Q = £/max of conjoined bases (X, U) with eventually invertible matrix X(t), i.e., to Rg(t) = I on [a, oo). Therefore, the present study can be regarded as a generalization and completion of the theory of the Riccati equations (R) for completely controllable systems (H) using the minimal genus Q = Qm-ln, as well as the noncontrollable systems (H) using the maximal genus Q = £/max- Among other new results in this paper (Theorem 6.3 and Corollary 6.4) we mention a connection of the symmetric solutions Q(t) of (7?.) with the implicit Riccati equation Re(t) [Q' + QA(t) + AT(t) Q + Q B(t) Q - C(t)} Rg(t) = 0. (1.6) Such implicit Riccati equations occur in the study of nonnegative quadratic functional associated with system (H), see [13, Section 6]. The study of the Riccati equations in the context of the present paper is also motivated by several situations in the literature, which are equivalent to using the Riccati matrix differential equation for an uncontrollable linear Hamiltonian system. For example, in [35, pg. 886], [1, pp. 621-622], [11, Sections 4 and 6], and [12, pp. 17-18] the authors use a cascade system of three differential equations for the investigation of calculus of variations or optimal control problems with variable endpoints - the Riccati equation (R), a linear differential equation, and an integrator. These three differential equations are together equivalent to a Riccati equation in dimension 2n, which corresponds to an uncontrollable system (H) in dimension An. This connection is discusses in details in [11, Remark 6.3]. The results of this paper open new directions in the theory of Riccati matrix differential equations associated with general uncontrollable linear Hamiltonian systems. They demonstrate that, as in the completely controllable case, distinguished solutions at infinity play a prominent role in the structure of the space of symmetric solutions of (1Z). Moreover, the intimate connection with the principal solutions of (H) at infinity points to effective applications of the distinguished solutions of (TV) at infinity in other fields of mathematics and engineering. The paper is organized as follows. In Section 2 we display the notation and preliminary results about system (H) and its solutions. In Section 3 we present properties of principal solutions of (H) at infinity and recall the concept of a genus of conjoined bases of (H). In Section 4 we develop the theory of Riccati differential equations for a given genus Q. In Section 5 we study inequalities for Riccati type quotients associated with the Riccati equation (TV). In Section 6 we analyze the relationship between the two Riccati equations (TV) and (1.6). In Section 7 we define 1688 peter šepitka the notion of a distinguished solution of (TV) at infinity and study its minimality properties. Finally, in Section 8 we provide examples illustrating our new theory. 2. Preliminaries about linear Hamiltonian systems. In this section we review some recent results about linear Hamiltonian systems (H) from [f 8, 33, 28, 29, 30, 31]. For a general theory of these systems we refer to [7, 17, 22]. By a matrix solution of (H) we mean a pair of functions (X, U) such that X, U : [a, oo) —> R™x™ are piecewise continuously differentiable (C*) and satisfy system (H) on [a, oo). In order to shorten the notation and the calculations, we sometimes suppress the argument t in the solutions. For any two matrix solutions (X\, U\) and (X2, U2) of (H) their Wronskian XfU'z — Uj X2 is constant on [a, 00). A solution (X, U) of (H) is called a conjoined basis if rank (XT (t), UT(t))T = n and XT(t) U(t) is symmetric at some and hence at any t € [a, 00). The principal solution (Xa, Ua) at the point a € [a, 00) is defined by the initial conditions Xa(a) = 0 and Ua(a) = I. By [17, Corollary 3.3.9], a given conjoined basis (X, U) can be completed to a fundamental system of (H) by another conjoined basis (X, U). In addition, the conjoined basis (X, U) can be chosen so that (X, U) and (X, U) are normalized, i.e., we have XTU - UTX = I. (2.1) The oscillation of conjoined bases of (H) is defined via the concept of proper focal points, see [36, Definition 1.1]. However, this concept will not be explicitly needed in this paper. By [33, Definition 2.1], a conjoined basis (X,U) of (H) is called nonoscillatory if there exists a € [a, 00) such that KerX(t) is constant on [a, 00). The main result of [33] then describes the nonoscillatory behavior of system (H), see Proposition 2.f below. Based on this result we say that system (H) is nonoscillatory if one and hence all conjoined bases of (H) are nonoscillatory. Proposition 2.1. Assume that the Legendre condition (1.1) holds. Then there exists a nonoscillatory conjoined basis of (H) if and only if every conjoined basis of (H) is nonoscillatory. For a subspace V C R™ we denote by V\> the orthogonal projector onto V. That is, V\> is a symmetric and idempotent nxn matrix such that ImV\> = V = Ker(/ — V\>) and KerVv = V1" = Im(/ — Vv). Orthogonal projectors can be constructed by using the Moore-Penrose pseudoinverse. More precisely, for a given matrix M € R™x™ and its pseudoinverse the matrix MM^ is the orthogonal projector onto ImM, and the matrix M is the orthogonal projector onto ImM^ = ImMT. Moreover, rankM = rank MM1" = rank M^M and Ker(MN) = Ker(MtM7V) for any matrices M,N € R™x™. For the theory of pseudoinverse matrices we refer to [4], [5, Chapter 6], and [6, Section 1.4]. In particular, we will need the following results on the differentiability of the Moore-Penrose pseudoinverse of a matrix-valued function M(t). Remark 2.2. By [6, Theorems 10.5.1 and 10.5.3], for a differentiable matrix-valued function M(t) on an interval [a, 00) its Moore-Penrose pseudoinverse (t) is differentiable on [a, 00) if and only if rankM(t) is constant on [a, 00). In this case (suppressing the argument t) (M1")' = -M^M'M^ + (I-M^M)(M')TM^TM^+M^M^T(M')T(I-MM^) (2.2) riccati equations for linear hamiltonian systems 1689 on [a, oo), see also [28, Remark 2.3]. Moreover, when KerM(i) is constant on [a, oo), then we have Ker M(i) C Ker M'(£) on [a, oo) and hence (2.2) reduces to (M^)'(t) = -M\t) M'(t) M\t) + M\t) MtT(t) (M')T(t) [I - M(t) M^(t)} (2.3) for every t € [a, oo). In particular, when the matrix M(t) is symmetric and Ker M(t) is constant on [a, oo), then (2.3) yields the standard formula (M^)'(t) = -M\t) M'(t) M\t), t e [a, oo). In the rest of this section (except of Theorem 2.9) we present known properties of conjoined bases of (H) with the corresponding references to the literature. Given a conjoined basis (X, U) of (H), by the kernel, image, and rank of (X, U) we mean the kernel, image, and rank of the component X. On the interval [a, oo) we define the orthogonal projectors onto the subspaces ImXT(t) and lmX(t) by P(t) := VlmXT{t) = X\t) X(t), R(t) := PImX{t) = X(t) X\t). (2.4) If (X, U) has constant kernel on [a, oo) C [a, oo), then by (2.4) the function P(t) is constant on [a, oo) and we set P := P(t) on [a,oo). (2.5) In this case (X, U) has constant rank r on [a, oo) with r := rankX(t) = rankP = rankR(t) on [a, oo), (2-6) and hence it follows from Remark 2.2 that the function X^ € on [a, oo). Consequently, the Riccati quotient Q(t):=X(t)X\t)U(t)X\t) = R(t)U(t)X\t), te[a,oo), (2.7) is piecewise continuously differentiable on [a, oo) as well. In addition, by [25, pg. 24] the matrix Q(t) is symmetric and satisfies on [a, oo) the properties (suppressing the argument t) XTQX = XTU, ImQCImi?, QX = RU. (2.8) The next statement is proven in [28, Theorem 4.2 and Equation (4.8)]. We observe that the Legendre condition (1.1) is not needed in this case. Proposition 2.3. Let (X, U) be a conjoined basis of (H) with constant kernel on the interval [a, oo) C [a, oo) and let P, R(t), and Q(t) be the corresponding matrices in (2.5), (2.4), and (2.7). Then the equalities Im[U(t) (I - P)}= Ker R(t), B(t) = R(t) B(t) = B(t) R(t) (2.9) hold for all t € [a, oo). Moreover, the matrix R(t) solves the Riccati equation (1.2) on [a, oo), while X^ satisfies on [a, oo) the formula (X1")' = XUT(7 -R)- X^AR - X^BQ. (2.10) Following [28, Section 4], with any conjoined basis (X,U) of (H) with constant kernel on [a, oo) we associate the S-matrix as the matrix-valued function Sa(t):= [ X]{s)B(s)X]T(s)As, te[a,oo). (2.11) J a Under (1.1), the matrix Sa(t) is symmetric, nonnegative definite, Sa € on [a, oo), and by [28, Theorem 4.2] the set lmSa(t) is nondecreasing on [a, oo) and hence eventually constant with lmSa(t) C ImP. By the symmetry of Sa(t), the set Ker Sa(t) is nonincreasing on [a, oo) and hence eventually constant with KerP C Ker Sa(t). 1690 peter šepitka This implies that the orthogonal projector onto the set lmSa(t) is eventually constant and we write PsJt) := PimSa{t) = sa(t)si(t) = Si(t)Sa(t), Psaoo ■= PSa(t) fort^oo. In addition, on [a, oo) we have the inclusions lmSa(t) = ImPSo(t) C lmPSa00 C ImP. The main properties of the function Sa(t) are summarized in the following statement, which follows from the definition of Sa(t) in (2.11), Remark 2.2, and (1.1), see also [28, Theorem 6.1]. Proposition 2.4. Assume (1.1). Let (X,U) be a conjoined basis of (H) with constant kernel on [a, oo) and let Sa(t) be the corresponding matrix defined in (2.11). Then the matrix function Sa(t) is nondecreasing on [a, oo). Moreover, if Sa(t) has constant kernel on a subinterval X C [a, oo), then € Cp(Z) and S^(t) is nonin-creasing on I. In particular, if Sa(t) has constant kernel on I = [/3, oo), then the limit of (t) as t —> oo exists. Remark 2.5. Under (1.1), the results in Proposition 2.4 and the properties of the matrix function Sa(t) discussed above imply that for every conjoined basis (X, U) of (H) with constant kernel on an interval [a, oo) the limit Ta := lim Sl(t) (2.13) t—>oo is well defined and it is referred to as the T-matrix corresponding to the conjoined basis (X, U) on [a, oo). Moreover, the matrix Ta is symmetric, nonnegative definite, and lmTa C lmPSa00 by (2.12) and ImSt (f) = ImSj(i) = lmSa(t) on [a,oo). Remark 2.6. The matrix Sa(t) is intimately connected with a certain class of conjoined bases of (H) which are normalized with (X, U). As we showed in [28, Theorem 4.4], for a given conjoined basis (X, U) with constant kernel on [a, oo) there exists a conjoined basis (X, U) of (H) such that (X, U) and (X, U) are normalized, i.e., (2.1) holds, and X\a)X{a) = Q. (2.14) The matrices X(t), X(t) P, and U(t) P are uniquely determined by (X, U) on the interval [a, oo) and X(t)P = X(t)Sa(t), U(t) P = U(t) Sa(t) + X^T(t) + U(t) (I - P) XT(t) X^T(t) for every t € [a, oo), where P is given in (2.5), see [28, Remark 4.5.(ii)]. We also note that according to [28, Theorem 5.2] the matrix Sa(t) satisfies the identities Xa(t) = X(t)Sa(t)XT(a), X\t)Xa(t) = Sa(t)XT(a), te[a,oo), (2.16) where (Xa, Ua) is the principal solution of (H) at the point a. As it is common, see [23, Section 3] or [28, Section 5], we denote by A[a, oo) the linear space of n-dimensional vector-valued functions u € Cp, which satisfy the equations u' = —AT(t) u and B(t) u = 0 on [a, oo). The functions u € A [a, oo) correspond to the vector solutions (x = 0,«) of system (H) on [a, oo). The space A [a, oo) is finite-dimensional with d[a, oo) := dimA[a, oo) < n. The number d[a, oo) is called the order of abnormality of system (H) on the interval [a, oo). riccati equations for linear hamiltonian systems 1691 We remark that system (H) is said to be normal on [a, oo) if d[a, oo) = 0, while it is called identically normal (or completely controllable) on [a, oo) if d(I) = 0 for every nondegenerate subinterval I C [a, oo). Moreover, for a given t € [a, oo) we denote by At[a, oo) the subspace in R™ of values of functions u € A [a, oo) at the point t, i.e., At [a, oo) := {c e R™, u(t) = c for some u e A [a, oo)}, t e [a, oo). (2.17) It is easy to see that dimAt[a, oo) = d[a, oo) for all t € [a, oo). We note that the set A[t, oo) is nondecreasing in t on [a, oo) and hence it is eventually constant. This means that the integer-valued function d[t, oo) is nondecreasing, piecewise constant, and right-continuous on [a, oo). In particular, there exists the limit rfoo := lim d[t, oo) = max d[t, oo), 0 < rfoo < n, (2-18) t—>oo t€[a, oo) which is called the maximal order of abnormality of (H). The monotonicity of the function oo) justifies the existence of the point € [a, oo) such that «00 := minja € [a, oo), rf[a, oo) = d^}. (2-19) From (2.18) and (2.19) we then obtain that the subspace AfaocOo) satisfies A^oo, oo) = lim^oo A [a, oo) = maxae[ai00) A [a, oo), A [a, oo) = A[aoo, oo), a € [ctoo, oo). On the other hand, for any a € [a, oo) the subspace A[a,t] is nonincreasing in t on (a, oo) and hence it is eventually constant. In particular, the integer-valued function d[a,t] is nonincreasing, piecewise constant, and left-continuous on (a, oo), see also [28, Section 5]. Moreover, we get d[a, oo) = lim rf[a, t] = min d[a,t], (2-21) t—>oo t€(a,oo) A [a, oo) = lim A[a,t] = min A[a,t] (2.22) t^oo t£(a,oo) for all a € [a, oo). For any such a point a the relation in (2.21) and the above properties of the function d[a, t] yield the existence of the point raj ^ in the interval [a, oo) such that Ta, oo := inf{£ € (a, oo), d[a, t] = d[a, oo)}. (2.23) Remark 2.7. We note that the subspace A[a,t], resp. A[a, oo) is closely related with the matrix Sa(t) in (2.11). More precisely, under (1.1) for every conjoined basis (X, U) of (H) with constant kernel on [a, oo) the corresponding matrices Pga (t) and Psaoo in (2-12) satisfy ImX(a) PSa(t) = (Aa[a,t])J~, te (a,oo), ImX(a) PSa00 = (A„[a, oo))^. The proof of the first formula in (2.24) is based on (2.16) and [28, Equation 5.6] in which we showed that Aa[a,t] = KerXa(t) holds on [a, oo), where (Xa, Ua) is the principal solution at the point a. The second identity in (2.24) follows from the first one by using (2.12) and (2.22). Moreover, in [28, Theorem 5.2 and Remark 5.3] we proved the equalities lmSa(t) = lmPsa00, rank5„(t) = r&nkPsa00 = n — rf[a,oo) (2.25) on (raiOOJoo) with r„i00 defined in (2.23). 1692 peter šepitka Throughout this paper we will consider only the intervals [a, oo) with the maximal order of abnormality rfoo defined in (2.18). The next remark shows how this condition reflects the properties of 5-matrices corresponding to conjoined bases of (H) with constant kernel. Remark 2.8. (i) Assume (1.1) and let (X,U) be a conjoined basis of (H) with constant kernel on [a, oo). In [26, Theorem 4.1.12] we proved that the condition d[a, oo) = rfoo holds if and only if the matrix Sa(t) in (2.11) associated with (X, U) satisfies the equalities Im[PSc>oo - Sa(t)Ta] = lmPSa0o = lm[PSaoo - Sa(t)Ta]T (2.26) for every t € [a, oo), where Psaoo and Ta are corresponding matrices in (2.12) and (2.13). We note that the identities in (2.26) can be equivalently replaced by rank [ Psaoo — Sa(t) Ta] = n — d[a, oo) on [a, oo), see [28, Theorem 6.9]. In addition, by [28, Equation 5.13] the conjoined basis (X, U) satisfies the conditions n — rfoo = n — d[a, oo) < rankX(t) < n for all t € [a, oo). (2.27) (ii) Let T/3 be the T-matrix in (2.13), which is associated with (X,U) on the interval [/3, oo) for /3 € [a, oo). In [26, Theorem 4.3.1(h)] we showed that the set IvnT/s is constant in /3 on [a, oo) if and only if the condition d[a, oo) = rfoo holds. The following theorem is an extension of the result presented in Remark 2.8(i). Theorem 2.9. Assume (1.1) and let (X,U) be a conjoined basis of (H) with constant kernel on [a, oo). Moreover, let P, Sa(t), and Ta be its corresponding matrices in (2.5), (2.11), and (2.13), respectively. Then the condition d[a, oo) = rfoo is equivalent with the formulas Im[P - Sa(t)Ta] =lmP = lm[P - Sa(t)Taf, te[a,oo). (2.28) Proof. First we remark that by PTa = Ta and PSa(t) = Sa(t) on [a, oo) we always have the inclusions Im[P-Sa(t)Ta] C ImP, lm[P-Sa(t)Taf = lm[P-Ta Sa(t)} C ImP (2.29) for every t € [a, oo). And since rank [P — Sa(t)Ta] = rank [P — Sa(t)Ta]T for all t € [a, oo), it is sufficient to show the equivalence of d[a, oo) = rfoo and the equality Im [P — Sa(t)Ta] = ImP on [a, oo). Assume that d[a, oo) = rfoo holds. Fix t e [a, oo) and let v e Ker [P - Sa{t)Ta]T = Ker [P - TaSa(t)}, that is, [P — Ta Sa(t)] v = 0. Using the latter equation and the identities Psaoo P = PsQoo and PsQOO Ta = Ta yields [PSq00 - Ta Sa(t)] v = PSaOC1 [P - Ta Sa(t)} v = 0 and hence, the vector v € Ker [Psa0o — Ta Sa(t)] = Ker [Psa0o — Sa(t)Ta]T = KerPsa0o, by the first equality in (2.26). Moreover, with the aid of identity Sa(t) = Sa(t) Psaoo we then get Pv=[P- Ta Sa(t)] v + Ta Sa(t) v = [P-Ta Sa(t)] v + Ta Sa(t) PSa00 v = 0, which shows that v € Ker P. Therefore, the inclusion Ker [P — Sa (t) Ta]T C Ker P, or equivalently, the inclusion ImP C Im [P — Sa(t)Ta] holds for every t € [a, oo). Combining the latter relation with the first property in (2.29) gives the equality Im[P - Sa(t)Ta] = ImP on [a,oo). Conversely, if Im [P - Sa(t)Ta] = ImP is satisfied for all t € [a, oo), then Im [ PSaoo ~ Sa(t) Ta] = Im PSa00 [ P ~ Sa(t) Ta] = Im PSq00 P = ImP5a00 riccati equations for linear hamiltonian systems 1693 on [a, oo), showing the first identity in (2.26). Finally, the condition d[a, oo) = rfoo holds by Remark 2.8(i), which completes the proof. □ The next statement is a combination of [28, Theorem 4.6] and [26, Theorem 2.3.3]. We again note that the Legendre condition (1.1) is in this case not needed. Proposition 2.10. Let (X, U) be a conjoined basis of (H) with constant kernel on [a, oo) and let P be its corresponding orthogonal projector in (2.5). Moreover, let (X, U) be a solution of (H), which is expressed in terms of (X, U) via matrices M,N € Rnxn, that is, where (X,U) is a conjoined basis of (H) satisfying (2.1) and (2.14) with regard to (X, U). Then the inclusion Iml(a) C Iml(a) holds if and only if ImN C ImP. In this case the matrices M and N do not depend on the particular choice of (X, U). In addition, if (X, U) is a conjoined basis with constant kernel on [a, oo) and the equality Iml(a) = Iml(a) holds, then M is nonsingular, MTN = NTM, Im7VTCImP, (2.31) where P is the matrix in (2.5) associated with (X, U). Remark 2.11. (i) Combining (2.14) with formulas (2.15) and (2.30) at t = a we obtain that the solutions (X, U) and (X, U) in Proposition 2.10 satisfy, see also [26, Equation (2.52)], X(a) = X(a)M, U(a) = U(a) M + X]T{a) N, ImTVCImP. The first equality in (2.15) allows to rewrite the expression for the matrix X(t) in (2.30) into the form X(t) = X(t)[M + Sa(t)N]=X(t)[PM + Sa(t)N] on[a,oo), (2.32) where Sa(t) is the 5-matrix in (2.11) associated with (X,U). In particular, this shows that the inclusion lmX(t) C lmX(t) holds for every t € [a, oo), see [26, Theorem 2.3.3]. We also note that the matrix TV is the (constant) Wronskian of (X, U) and (X, U). (ii) Let (X, U) be a conjoined basis of (H) with constant kernel on [a, oo) such that Iml(a) = Iml(a) holds. Then lmX(t) = lmX(t) on [a, oo), as we proved in [28, Theorem 4.10]. If Sa(t) is the 5-matrix which corresponds to (X, U) on [a, oo), then we have identities [PM + Sa(t) TV]1" = PM'1 - Sa(t) NT, (2.33) Im [PM + Sa(t) N] = ImP, Im [PM + Sa(t) N]T = Im P, (2.34) Sa(t) = [PM+ Sa{t)N]] Sa{t)MT-1P, lmSa(t) =lmPM-1Sa(t) (2.35) for every t € [a, oo), see [28, Remark 4.7, Theorem 4.10]. In particular, since Sa(a) = 0 = Sa(a) by (2.11), formula (2.33) at t = a and the inclusion in (2.31) give the equalities (PM)] = PM'1, N{PM)] = NPM'1 = NM'1. (2.36) 1694 peter šepitka Moreover, the identities in (2.32) and (2.34) yield X(t) = X(t) [PM + Sa(t) TV]1" and the formulas for the pseudoinverses X\t) = [PM + Sa(t)N]l X\t), Xt(i) = [PM + Sa(t)N] Xt(i), on [a,oo). (2.37) 3. Principal solutions at infinity. Following [29, Definition 7.1], we say that a conjoined basis (X, U) of (H) is a principal solution at infinity if (X, U) has constant kernel on [a, oo) and its corresponding matrix Sa(t) defined in (2.11) through X(t) satisfies (t) —> 0 as t —> oo, that is, Ta = 0 in (2.13). In this case we will say that (X, U) is a principal solution of (H) at infinity with respect to the interval [a, oo). By (2.27), the principal solutions of (H) can be classified according to the rank of X(t) on [a, oo). In particular, the minimal principal solution (Xm-ln, Um-ln) of (H) at infinity satisfies rankXm;n(t) = n — d^, while the maximal principal solution (Jmax, t/max) °f (H) at infinity is determined by rankXmax(t) = n, hence Xmax(t) is invertible on [a, oo), see [29, Remark 7.2]. In the next proposition we recall from [29, Theorem 7.6] and [28, Theorems 7.6] the characterization of the nonoscillation of system (H) by the existence of a principal solution of (H) at infinity with any possible rank, as well as the uniqueness of the minimal principal solution. Proposition 3.1. Assume that (f.f) holds. Then the following statements are equivalent. (i) System (H) is nonoscillatory. (ii) There exists a principal solution of (H) at infinity. (iii) For any integer r satisfying n — d^ < r < n there exists a principal solution of (H) at infinity with rank equal to r. In particular, system (H) is nonoscillatory if and only if there exists a minimal principal solution of (H) at infinity. In this case the minimal principal solution is unique up to a right nonsingular constant multiple. In [29, Equation 7.4] we defined for a nonoscillatory system (H) the point am-ln € [a, oo) by "mm := inf € [a, oo), (Xm;n, J7min) has constant kernel on [a, oo)}, (3.1) where (Xm-ln, Um-ln) is the minimal principal solution of (H) at infinity. We note that the equality d[a, oo) = rfoo holds for every a > am-ln, see [29, Theorem 7.9]. In turn, combining this fact with formula (2.19) we obtain that ^[^minj Oo) d^, i.e., Amin ^ ^oo- The next results are based on [28, Theorem 7.5] and [29, Lemma 7.5 and Remark 7.11]. Proposition 3.2. Assume that (1.1) holds and system (H) is nonoscillatory with "mm defined in (3.1). Then the following statements hold. (i) If (X, U) is a principal solution of (H) at infinity with respect to the interval [a, oo), then d[a, oo) = d^. Moreover, the pair (X, U) is a principal solution of (H) at infinity also with respect to the interval [/3, oo) for every /3 > a. riccati equations for linear hamiltonian systems 1695 (ii) Every principal solution (X, U) of (H) at infinity is a principal solution with respect to [a, oo) for every a € (amm, oo). In other words, the conjoined basis (X, U) has constant kernel on the open interval (amin, oo) and its corresponding matrix Sa(t) in (2.11) satisfies S^(t) —> 0 as t —> oo for every a > am;n. Remark 3.3. We note that the orthogonal projector Pg x in (2.12) associated with the principal solution (X,U) through the matrix Sa(t) is the same for all initial points a € (am;n,oo), see [29, Remark 7.11]. Therefore, we will use the notation PSoo := PSaoo fOT a 6 ("min, Oo). (3.2) Given a principal solution (X, U) of (H) at infinity, we define the point a € [a, oo) associated with (X, U) by a := inf {a € [a, oo), (X, U) is a principal solution ^ of (H) with respect to [a, oo)}. From Proposition 3.2 it immediately follows that the point a in (3.3) satisfies the inequalities < a < am-ln with defined in (2.19). We also note that the set (a, oo) is the maximal open interval with the property that (X, U) is a principal solution of (H) with respect to [a, oo) for every a € (a, oo). Therefore, we will often say that (X, U) is a principal solution of (H) at infinity with respect to the maximal interval (a, oo). In particular, the conjoined basis (X,U) has constant kernel on the open interval (a, oo) and the 5-matrix Sa(t) in (2.11) associated with (X, U) satisfies (t) —> 0 as t —> oo for every a > a. In the next theorem we derive an exact relation between the points a and am-ln. Theorem 3.4. Assume that (1.1) holds and system (H) is nonoscillatory with am-ln defined in (3.1). Let (X,U) be a principal solution of (H) at infinity and let a be its corresponding point in (3.3). Then the equality a = am-ln holds. Proof. Let (X, U), a, and am;n be as in the proposition and suppose that a < am-ln. According to (3.3) there exists a point ft € (a, amm) such that (X, U) is a principal solution of (H) at infinity with respect to the interval [/3, oo). By Proposition 3.2(i) with a := ft we know that d[/3, oo) = d^. Let (Xm-ln, Um-ln) be the minimal principal solution of (H) at infinity. By [29, Theorem 7.3] it follows that the pair (Xm-ln, Um-ln) is a minimal principal solution at infinity with respect to the interval [/3, oo). For this we note that (Xm-ln, Um-ln) is contained in (X, U) on [/3, oo) according to the properties of the relation "being contained" in [28, Section 5]. The uniqueness of the minimal principal solution and the definition of am;n in (3.1) then yield that P > "mm, which is a contradiction. Therefore, the equality a = am-ln holds and the proof is complete. □ In the following result we present a construction of a principal solution of (H) at infinity from a conjoined basis of (H) with constant kernel on [a, oo). It is a generalization of [32, Equation (10)], where only the minimal principal solution of (H) was considered. This result will be utilized for the construction of a distinguished solution of (TV) at infinity in Theorem 7.16. Theorem 3.5. Assume that condition (1.1) holds and system (H) is nonoscillatory. Let a € [a, oo) be such that d[a, oo) = rfoo and let there exists a conjoined basis of 1696 peter šepitka (H) with constant kernel on [a, oo). Then a solution (X,U) of (H) is a principal at solution infinity with respect to the interval [a, oo) if and only if for some conjoined basis (X,U) of (H) with constant kernel on [a, oo). Here the conjoined basis (X, U) and the matrix Ta are associated with (X, U) through Remark 2.6 and (2.13). Proof. If (X, U) is a principal solution of (H) at infinity with respect to [a, oo), then (X, U) has constant kernel on [a, oo) and the associated matrix Ta in (2.13) satisfies fa = 0. Formula (3.4) then holds trivially with (X,U) := (X,U). Conversely, let (X, U) be a conjoined basis of (H) with constant kernel on [a, oo) and let P, Sa(t), Psaoo, and Ta be the matrices in (2.5), (2.11), (2.12), and (2.13) corresponding to (X,U) on [a, oo). Consider the matrix solution (X,U) of (H) in (3.4). Since PTa= Ta, it follows from Proposition 2.10 with M := I, N := -Ta and (X, U) := (X, U) that (X, U) is a conjoined basis of (H), which in turn by (2.32) yields that X(t) = X(t) [P - Sa(t)Ta] on [a,oo). Then, by PSa(t) = Sa(t) and using (2.28) from Theorem 2.9, we get KerX(i) = Ker X\t)X{t) [P - Sa{t)Ta] = Ker P [P - Sa{t)Ta] = Ker [P - Sa(t) Ta] (2=} (Im P)1- = Ker P on [a, oo). This shows that (X, U) has constant kernel on [a, oo) as well. Moreover, if P, Sa(t) and Pg x are the matrices in (2.5), (2.11), and (2.12) corresponding to (X, U) on [a, oo), then by using the first equations in (2.36) and (2.35), respectively, we have that P = P and Sa(t) = [P- Sa(t) Ta]t Sa(t) P=[P- Sa(t) Ta]t Sa(t) (3.5) for all t € [a, oo). On the other hand, applying the second identity in (2.35) yields the equalities lmSa(t) = lmPSa(t) = lmPSa(t) = lmSa(t) on [a, oo), which in particular by (2.12) imply that Pj ^ = Psaoo- Let r„i00 be defined in (2.23). Then Sa(t)SUt) = PSaoo = 5t(t) Sa(t) and Sa(t) Sl(t) = PSaX = Sl(t) Sa(t) on (Taj00,oo), by (2.25). Consequently, with the aid of (2.28) and (3.5) together with the identity Ta Psaoo = Ta = Psa00 Ta we obtain that si(t) = pSa00 si(t) = si(t) sa(t) si(t) = si(t) psa(t) si(t) { =8) SJ,(t) [P - Sa(t)Ta] [P - 5a(t)Ta]t Sa(t)Sl(t) (=5) St (t) [P - Sa(t) Ta] Sa(t) Sl(t) = Sl(t) [P - Sa(t) Ta] PSa00 = St(t) [P - Sa(t) Ta] PSa00 = Si(t) - Ta (3.6) for every t G (Tat00,oo). Finally, formula (3.6) implies that 5*t(t) —> 0 as t —> oo. This shows that the conjoined basis (X, U) is a principal solution of (H) at infinity with respect to the interval [a, oo). □ Remark 3.6. It follows from Proposition 2.f0 and Remark 2.11 (i) that the principal solution (X,U) constructed in (3.4) satisfies the equality Iml(a) = Iml(a). riccati equations for linear hamiltonian systems 1697 Moreover, as we noted in Remark 2.11 (ii), this condition is valid on the whole interval [a, oo), i.e., ImA(i) = ImX(t) holds for all t € [a, oo). In particular, the last equality means that the conjoined bases (A, U) and (X, U) belong to the same genus of conjoined bases of (H) as we define below, see also Remark 3.13. In the second part of this section we recall basic concepts from the theory of genera of conjoined bases of (H) from our recent work [27, Section 4]. We wish to point out that in this context the Legendre condition (1.1) is not assumed and/or system (H) is allowed to be oscillatory. Define the orthogonal projector RAoa(t) := Vwj-, where Wt := A^a^, oo), te [«00,00), (3.7) where the point is determined in (2.19) and the subspace At[aoo, 00) is defined in (2.17). From the second identity in (2.20) it follows that for any a > the matrix RAoo(t) defined in (3.7) is the orthogonal projector onto the set (At[a, 00)) on [a, 00), i.e., RAoa(t) = Tu±, where Ut := At[a, 00), t e [a, 00). (3.8) Remark 3.7. Assume the Legendre condition (1.1). Let (X, U) be a conjoined basis of (H) with constant kernel on [a, 00) C [a^, 00) and let P and Psaoo be the corresponding orthogonal projectors in (2.4) and (2.12), respectively. Combining (2.24) and (3.8) then yields the identity ImX(a) PSaOC = lmRAoo(a). (3.9) Moreover, since Im [X(a) PSaoo]T = Psaoo, PPsaoo = Psaoo, and X^(a)X(a) = P, we have that [X(a) PSct00]t = PPSa00 [A» PSct00]t = At (a) X(a) PSa00 [A» PSa00}^ = X\a)[X(a)PSaoo}[X(a)PSaoo}i { = ]X\a)RAao(a). (3.10) The orthogonal projector RAoo(t) defined in (3.7) plays a crucial role in the following notion. According to [27, Definition 4.3] we say that two conjoined bases (Ai, U\) and (A2, U2) of (H) have the same genus (or they belong to the same genus) if there exists a e [«00, 00) such that ImAi(t) + lmRAoa(t) = Im X2(t) + lmRAoa(t), te [a, 00). From this definition it follows that the relation "having (or belonging to) the same genus" is an equivalence on the set of all conjoined bases of (H). Therefore, there exists a partition of this set into disjoint classes of conjoined bases of (H) with the same genus. This allows to interpret each such an equivalence class Q as a genus itself. The following result is proven in [27, Theorem 4.5]. Proposition 3.8. Let (Xi,U\) and (Ag,^) be conjoined bases of (H). Then the following statements are equivalent. (i) The conjoined bases (Xi,U\) and (Ag,^) belong to the same genus. (ii) The equality Im A(t) + lvnRAao(t) = ImA2(i) + lvnRAao(t) is satisfied for every t € [«00, 00). (iii) The equality Im A(t) + lvnRAao(t) = ImA2(i) + lvnRAao(t) is satisfied for some t € [«00, 00). Let Q be a genus of conjoined bases of (H) and let (A, U) be a conjoined basis belonging to Q. The results in Proposition 3.8 imply that for all t € [«00,00) the 1698 peter šepitka subspace lmX(t) + lmRAoa(t) does not depend on the particular choice of (X, U) in Q. Therefore, the orthogonal projector onto lmX(t) + lmRAoa(t), i.e., the matrix Rg(t):=VVt, where Vt := ImX(t) + lmRAoa(t), te [«00,00), (3.11) is uniquely determined for each genus Q. The next statement is from [27, Theorem 4.7]. Proposition 3.9. Let Q be a genus of conjoined bases of (H) and let Rg(t) be the orthogonal projector defined in (3.11). Then the matrix Rg(t) is a solution of the Riccati equation (1.2) on [«00,00) and the inclusion lvnRAao(t) C lmRg(t) holds for every t € [«001 °°)- Remark 3.10. If the orthogonal projector Rg(t) satisfies Rg(t) = RAao(t) on [«00, 00), then the genus Q = Qm-ln is called minimal, while if Rg(t) = I on [«00, 00), then the genus Q = £/max is called maximal. The next result describes important properties of nonoscillatory conjoined bases from a given genus Q. These properties will be utilized in Section 4 to show their connection with symmetric solutions of the Riccati equation (7?.) associated with the genus Q, see Theorem 4.18. Proposition 3.11. Let Q be a genus of conjoined basis of (H) with the corresponding orthogonal projector Rg(t) in (3.11). Furthermore, let (X,U) be a conjoined basis of (H) with constant kernel on [a, 00) C [«00,00) such that (X,U) belongs to Q and let R(t) and Q(t) be the matrices in (2.4) and (2.7). Then the equality Rg(t) = R(t) holds for all t € [a, 00). Moreover, the matrices X(t), X^(t), and U(t) satisfy on [a, 00) the equations X' = (A + BQ)X, (X1")' = -X\A + BQ), (3.12) U' = AU+[C-(A + AT)Q]X, (3.13) where the matrices A(t) and B(t) are defined in (1.3). Proof. For the proof of Rg(t) = R(t) we refer to [27, Proposition 4.16]. We will prove that (3.12) and (3.13) hold. From the definition of the matrix A(t) in (1.3) it follows that A{t)Rg(t) = A(t) Rg(t) on [a, 00). Moreover, using (1.3), (2.8), (2.9), and the identity Rg(t) X(t) = X(t) on [a, 00) yields the formula X' {2= ARgX + BRU {2= ARgX + BQX {1= (A + BQ)X on [a, 00). Since the function X^ € Cp, equation (2.10) in Proposition 2.3 becomes (X1")' = -X*[ARg - AT{I-Rg)] -X^BQ (=3) -X\A + BQ) on [a, 00). Finally, by using Rg(t) U(t) = Q(t) X(t) for every t € [a, 00) we get U' - AU {1= CX - ATU - [ARg - AT(I - Rg)] U = CX -(A + AT) RgU ('=] CX -(A + AT) QX=[C-(A + AT) Q] X on [a, 00). Thus, the matrix U(t) solves (3.13) on [a, 00). The proof is complete. □ Remark 3.12. Let $a(i) be the fundamental matrix of the system Y' = [A(t) + B(t)Q(t)]Y for t € [a, 00) satisfying $„(«) = I. It is well-known that $j_1(t) is the fundamental matrix of the adjoint system Y' = —[A(t) + B(t) Q(t)]T Y for t € [a, 00). From (3.12) we then obtain by the uniqueness of solutions that X(t) = <&a(t)X(a), X\t)=X\a)<&-1(t), te[a,oo). (3.14) riccati equations for linear hamiltonian systems 1699 Remark 3.13. In [29, Theorem 7.12] we proved that every genus Q of conjoined bases of nonoscillatory system (H) contains a principal solution of (H) at infinity. Moreover, in Theorem 3.5 we described the construction of any such a principal solution in terms of conjoined bases from the genus Q, see also Remark 3.6. In the next proposition we recall from [29, Theorem 7.13] a complete classification of all principal solutions of (H) at infinity within the genus Q. Proposition 3.14. Assume that (1.1) holds and system (H) is nonoscillatory with "mm defined in (3.1). Let (X,U) be a principal solution of (H) at infinity, which belongs to a genus Q. Moreover, let P and Pg^ be the orthogonal projectors defined through the function X(t) on (am;n,oo) in (2.5), (2.12), and 3.2. Then a solution (X, U) of (H) is a principal solution belonging to Q if and only if for some (and hence for every) a € (amin, oo) there exist matrices M, N € Wnxn such that X(a) = X(a)M, U(a) = U(a) M + X^T(a) N, M is nonsingular, MTN = NTM, ImN C ImP, PSoo NM~1Pgoo = 0. 4. Riccati matrix differential equation for given genus. In this section we present a new theory extending the results by Reid in [22, 23] about Riccati matrix differential equation (R) to general possibly uncontrollable systems (H). Namely, for every genus Q of conjoined bases of (H) we consider the Riccati matrix differential equation (TV). In Lemma 4.1, Theorem 4.3, and Corollary 4.5 we first derive properties of solutions of (TV) in the relation with the associated projector Rg(t) in (3.11). In (4.9) and (4.18) we introduce an auxiliary linear differential system and the so-called F-matrix for a solution of this system, which serve as main tools for the formulation of the results in this section. In particular, in Theorem 4.16 we present additional properties of solutions of (TV) obtained through the above mentioned F-matrix. The main results concerning the correspondence between the solutions of the Riccati equation (TV) and conjoined bases of (H) from the genus Q are contained in Theorems 4.18 and 4.21. First we derive some auxiliary properties of the projector Rg(t), being a solution of the Riccati equation (1.2), and the coefficient A(t) in (1.3). In particular, we represent Rg(t) as a solution of a linear differential system Rg = [A(t], Rg] = A(t) Rg - Rg A(t). (4.1) We note that since Rg(i) is symmetric, then it solves also the system Rg = [Rg, AT(t)] = Rg AT(t) - AT(t) Rg. (4.2) being the transpose of the system in (4.1). For M, N € R™x™ the notation [M, N] used in (4.1) and (4.2) means their commutator, i.e., [M, N] := MN — NM. Lemma 4.1. Let Q be a genus of conjoined bases of (H) and let Rg(t) and A(i) be the corresponding matrices in (3.11) and (1.3). Then [Rg(t), A(t)+AT(t)] = 0 for every t € [«00,00), i.e., the matrices Rg(t) and A(t) + AT(t) commute on [«00,00). Moreover, the orthogonal projector Rg(t) satisfies on [«00,00) system (4.1). Proof. First we note that from the definition of the matrix A(i) in (1.3) we have on [«oo,oo) the formulas ARg = ARg, RgAT = RgAT, (4.3) RgA = RgARg - RgAT(I - Rg) = Rg (A + AT) Rg - RgAT. (4.4) 1700 peter šepitka By combining equality (4.4) with the second identity in (4.3) we obtain that /.'.. (A + AT) = 11... \ + 1 ,>...{' = 1,'.. (A + AT) 11.. - 11.. A1 + 11.. A1 = Rg (A + AT) Rg (4.5) on [«00,00). From (4.5) it then follows that the matrix Rg(t) [A(i) + AT(t)] is symmetric for every t € [«oo, oo), which in turn implies the equality Rg(t) [A(t) + AT(t)} = [A(t) + AT(t)} Rg(t) for all t € [«00,00). In particular, this means that the matrices Rg(t) and A(t) + AT(t) commute on [«00,00), i.e., the commutator [Rg(t), A(i) + AT(t)] = 0 for every t € [«00, 00), showing the first part of the lemma. For the proof of the second part we note that the orthogonal projector Rg(t) solves the Riccati equation (1.2) on [«00,00), by Proposition 3.9. Moreover, by using formula (4.4) and the first identity in (4.3) equation (1.2) reads on [«00,00) as Rl {1=2) ARg - [Rg (A + AT) Rg - RgAT] {4'3)={4-4) ARg - RgA = [A, Rg}. Thus, the matrix Rg(t) solves system (4.1) on [«00,00)- □ Remark 4.2. We remark that the formulas in (4.f) are equivalent with (I — Rg)' = [A(t), I — Rg] = [I — Rg, AT(t)], te[«oo,oo), (4.6) as one can easily check. The matrix Rg(t) satisfies on [«oo, 00) also the relations Rg(t) = A(t) Rg(t) + Rg(t)AT(t) - Rg(t) [A(t) + AT(t)} Rg(t), lmB(t) C lmRg(t). Note that the first equation in (4.7) is the same as (f .2) with A(t) instead of A(t). Next we derive properties of the solutions of (7?.), which are based on the projector Rg(t) and Lemma 4.f. Theorem 4.3. Let Q be a genus of conjoined bases of (H) with the corresponding matrix Rg(t) in (3.11) and let Q(t) be a solution of the Riccati equation (TV) on [a, 00) C [«00,00). Then the matrices Rg(t)Q(t), Q(t)Rg(t), and Rg(t) Q(t) Rg(t) also solve (TV) on [a, 00). Proof. Let Rg(t) and Q(t) be as in the theorem. By using (7?.), the second formula in (4.1), and the identities Rg(t) C(t) = C(t) and B(t) Rg(t) = B(t) for every t e [a, 00) we obtain that (Re Q)' = RgQ + RgQ' {4Ak{n) [Rg,AT] Q + Rg (C - Q A - ATQ -QBQ) = Rg ATQ - ATRg Q + C-RgQA-Rg ATQ - RgQBRgQ = —AT(Rg Q) - (Rg Q)A- (Rg Q) B (Rg Q)+C on [a, 00). Thus, the matrix Rg(t) Q(t) solves (7?.) on [a, 00). Similarly, by the first formula in (4.1) and the identities C(t) Rg(t) = C(t) and Rg(t) B(t) = B(t) for all t € [a, 00) we get (QRg)' = Q'Rg + QRg {TC)^41) (C - Q A - ATQ - QBQ)Rg + Q[A, Rg] = C-Q ARg - ATQRg - QRg B QRg + Q ARg - QRgA = -AT(QRg) - (QRg) A - (QRg) B (QRg) + C riccati equations for linear hamiltonian systems 1701 on [a, oo), showing that the matrix Q(t) Rg(t) solves (TV) on [a, oo). Finally, by combining these results we get that also Rg(t) [Q(t) Rg(t)] = [Rg(t) Q(t)] Rg(t) = Rg(t) Q(t) Rg(t) is a solution of (7?.) on [a, oo) and the proof is complete. □ Remark 4.4. The symmetry of equation (7?.) implies that the matrix QT(t) solves equation (7?.) on [a, oo). By applying Theorem 4.3 for Q := QT we then obtain that also the matrices Rg(t) QT (t), QT(t) Rg(t), and Rg(t) QT(t) Rg(t) are solutions of (TV) on [a, oo). Corollary 4.5. With the assumptions and notations of Theorem 4-3, the matrix Q(t) satisfies the inclusion ImQ(t) C lmRg(t), resp. the inclusion lmQT(t) C lmRg(t), for all t € [a, oo) if and only if the inclusion ImQ(to) Q Im.Rs(£o)> resp. the inclusion ImQT(io) C ImRg(to), holds for some tg € [a, oo). Proof. From Theorem 4.3 and Remark 4.4 we know that the matrices Qt(t) := Rg(t) Q(t) and Qtt(t) := Rg(t) QT(t) solve equation (TV) on [a, oo). Fix to € [a, oo). If ImQ(io) Q Imi?g(t0), then the matrix Qt(t0) = Q(t0) and by the uniqueness of solutions of (TV) we obtain the equality Qt(t) = Q(t) on [a, oo). The latter identity means that the inclusion lvnQ(t) C lvnRg(t) holds for every t € [a, oo). By using the similar arguments the relation ImQT(to) ^ Im.Rs(£o) implies that Q**(to) = Q(to) and consequently, we have the equality Qtt(t) = Q(t) on [a, oo). Hence, the inclusion ImQT(i) C lmRg(t) holds for every t € [a, oo). The proof of opposite implications is trivial. □ For our reference we now present an auxiliary result from linear algebra about orthogonal projectors. Lemma 4.6. Let Z € R™x™ be an orthogonal projector. Then K, L € R™x™ satisfy ImK ClmZ and ImtCKerZ (4.8) if and only if K = ZE and L = (I — Z) E for some matrix E € R™x™. In this case the equality Ker_ftT n KerL = Ker_E holds. Proof. Let Z be as in the lemma. If the matrices K and L satisfy (4.8), then for E := K + L we have that ZE = ZK + ZL = K, (I - Z) E = (I - Z) K + (I - Z) L = L. The opposite implication is trivial. Finally, it is easy to see that in this case we have the equality Ker K n Ker L = Ker E, which completes the proof. □ Remark 4.7. Let Z be an orthogonal projector and let K and L be matrices satisfying (4.8). The results in Lemma 4.6 then show that Ker_ftT n KerL = {0} if and only if the matrix E = K + L is nonsingular. In particular, in this case the inclusions in (4.8) are implemented as equalities, i.e., the identities Im_ftT = ImZ and ImL = KerZ hold. We also note that the condition Ker_ftT n KerL = {0} is equivalent with rank(LTT, LT)T = n. Let Q be a genus of conjoined bases of (H) and let Rg(t) be its representing orthogonal projector in (3.11). For a given solution Q(t) of the Riccati equation (TV) on a subinterval [a, oo) C [«00,00) we consider the following system of first order linear differential equations 6' = [A(t)+B(t)Q(t)}0, Q' = A(t) Q + [I- Rg(t)]{C(t) - [A(t) + AT(t)} Q(t)} 9, 1702 peter šepitka on [a, oo) together with the initial conditions 9(a) = K, tt(a) = L, (4.10) where the matrices K, L € R™x™ satisfy ImK C im.Rs(a), Imt C Kerflg(a), rank (KT, LT)T = n. (4.11) We will study the properties of solutions of system (4.9), which will serve for the formulation and proofs of the main results of this section. The first equation in (4.9) is motivated by the approach in [23, Chapter 2, Lemma 2.1], which is adopted here to the setting of uncontrollable systems (H). Remark 4.8. Given a solution Q(t) of (TV) on [a, oo) we note that for any matrices K and L satisfying (4.11) there exist unique matrices 9(t) and fi(t), which solve the equations in (4.9) on [a, oo) with 9(a) = K and fl(a) = L. Moreover, in this case we have from Lemma 4.6 and Remark 4.7 with Z := Rg(a) that 9(a) = Rg(a) E and fl(a) = [I — Rg(a)] E for some nonsingular matrix E. These observations then imply that the initial value problem (4.9)-(4.10) with (4.11) has the solution (9, fl), which is unique up to a right nonsingular multiple. More precisely, if (9o,Slo) is another solution of (4.9)-(4.11), then there exists a constant nonsingular matrix M e Rnxn such that 90(t) = 9(t) M and fi0(i) = Cl(t) M for all t e [a, oo). Proposition 4.9. Let Q be a genus of conjoined bases of (H) with the corresponding matrix Rg(t) in (3.11). Moreover, letQ(t) be a solution of equation (TV) on [a, oo) C [000,00) and let (9, SI) be a solution of the associated system in (4.9) on [a, oo). Then the matrix 9(t) has a constant kernel on [a, oo) and the matrices V(t) := [I — Rg(t)]0(t) and W(t) := Rg(t)fl(t) solve on [a, oo) the linear differential equation Y' = A(t) Y. In addition, if the matrices K := 9(a) and L := fl(a) satisfy the conditions in (4.11), then for all t € [a, oo) we have Im6(t) = ImRg(t), Imfi(t) = KerRg(t), rank (9T(t), nT(t))T = n. (4.12) Proof. Let Rg(t), Q(t), 9(t), Sl(t), K, and L be as in the proposition. By the uniqueness of solutions of the first equation in (4.9) we have that 9(t) = Qa(t) 9(a), where Qa(t) is the associated fundamental matrix normalized at the point a, i.e., &a = [A(t) + B(t)Q(t)]a, te[a,oo), $a(a)=/. (4.13) This implies that Ker9(t) = Ker9(a) for every t € [a, oo), i.e., the matrix 9(t) has constant kernel on [a, oo). Next we show that the matrices V(t) and W(t) satisfy on [a, oo) the equation Y' = A(t)Y. Indeed, by using (4.1), (4.6), (4.9), and the inclusion in (4.7) we obtain on [a, oo) that V1 = (I - Rg)' 9 + (/ - Rg) 9' {4-6)={4-9) [A, I - Rg] 9 + (/ - Rg) (A + B Q) 9 {4=7) A(I - Rg) Q - (I - Rg)AO+ (I - Rg)AO = AV, (4.14) W = Rgn + Rg n' {zL1)={4-9) [A, Rg] il +Rg{An + (I - Rg)[C - (A + AT) Q] 9} = ARgtt-RgAtt + RgAtt = AW. (4.15) Moreover, suppose that the matrices K and L satisfy (4.11). Then V(a) = 0 = W(a), which in turn, by uniqueness of solutions of (4.14) and (4.15), implies that V(t) = 0 = W(t) for all t e [a, oo). Therefore, we have Im9(V) C lmRg(t) and Im fl(t) C Ker Rg (t) on [a, oo). And since the matrices 9(t) and Rg (t) have constant ranks on [a, oo) and the equality rank9(a) = ranki?s(a) holds by Remark 4.8, we riccati equations for linear hamiltonian systems 1703 obtain that even ImO(£) = lmRg(t) for every t € [a, oo). Now we shall prove the last condition in (4.12), which is clearly equivalent with the identity KerO(£) n Kerfi(t) = {0} on [a, oo). Fix /3 e [a, oo) and let v e KerO(/3) nKerfi(/3). From the fact that KerO(£) is constant on [a, oo) it then follows that Q(t)v = 0 for all t € [a, oo). In particular, this means that the function w(t) := fl(t) v satisfies on [a, oo) the identity w'(t) = A(t)w(t), by (4.9). But w(/3) = 0 and hence, by the uniqueness of solutions of the equation y' = A(t) y we get that w(t) = 0 for every t € [a, oo). Therefore, the vector v € KerO(a) n Kerfi(a) = Ker_ftT n KerL, which in turn implies that v = 0, by (4.11). Thus, the subspace Ker0(t)nKerfi(t) = {0} for all t € [a, oo). Finally, with the aid of Remark 4.7 we conclude that Imf2(i) = Keri?s(i) on [a, oo), showing the second condition in (4.12). □ Remark 4.10. The results in Proposition 4.9 and Remark 2.2 imply that for any solution (0,f2) of (4.9)-(4.11) the matrix 0^ € and satisfies the equation [0t(t)]' = _et(t) + B(t) Q(t) Rg(t)}, t e [a, oo). Moreover, from Remark 4.8 it follows that for a given solution Q(t) of the Ric-cati equation (TV) on [a, oo) C [«00,00) there exists the unique such a pair (0, Q,) satisfying 0(a) = Rg(a) and fl(a) = I — Rg(a). Obviously, in this case we have the equalities 0(t) = $>a(t) Rg(a) and Im9^(t) = ImOT(£) = lmRg(a) on [a, 00), where $a(i) is the fundamental matrix in (4.13). In particular, the matrix 0^(t) then satisfies for every t € [a, 00) the formula 0t(t) = Rg(a) 0t(t) = $a(t) Rg(a) ©t(f) = $-1(t)0(t)0tW (=2) KHt)Re(t). (4.16) On the other hand, by the aid of (4.12) we obtain the equality Ima(t) Rg(a) = Im0(£) (4=2) Im Re(t), te[a,oo). (4.17) This equality is an important property of the matrix function $>a(t), which will be utilized in the proof of Theorem 4.16 below. In the following remark we introduce an important matrix (called the F-matrix) in terms of a solution 0(t) of (4.9). For an invertible 0(t) this matrix was considered in [23, Section 2.2]. Here we allow 0(t) to be singular. Remark 4.11. (i) The properties of the matrix 0(t) allow to define the function Fa(t):= I 0t(s)S(s)0tT(s)ds, te[a,oo), (4.18) which will be referred to as the F'-matrix corresponding to the solution Q(t) with respect to the genus Q. From (4.18) it immediately follows that Fa(t) is symmetric and the inclusion lmFa(t) C lmRg(a) holds for every t € [a, 00) and Fa € Moreover, under (1.1) the matrix Fa(t) is nonnegative definite and nondecreasing with F'a(t) = ef(i) B(t) 0tT(t) > 0 on [a, 00). Therefore, the subspace KerF„(t) is nonincreasing on [a, 00), and hence eventually constant. Consequently, the properties of Moore-Penrose pseudoinverse displayed in Remark 2.2 imply that F^ € C* with (Ft)'(t) = -Ft(t)F^(t)Ft(t) = -Ft(t)et(t)B(t)6tT(t)Ft(t) < 0 for large t. Thus, the matrix F^(t) is nonincreasing for large t. And since F^(t) is nonnegative definite on [a, 00), it follows that the limit of F^(t) exists as t —> 00, i.e., Da := lim Fl(t). (4.19) 1704 peter šepitka Clearly, the matrix Da defined in (4.19) is symmetric and nonnegative definite and the inclusion lvnDa C lvnRG(a) holds. In addition, we note that with the aid of (4.16) and the identity Rg(t) B(t) Rg(t) = B(t) on [a, oo) the matrix Fa(t) in (4.18) can be also represented as Fa(t) ^1SU4-16) f^^R^B^R^^^ds \-\S)B(3)^l-1(S)d3 (4.20) for all t e [a, oo) with $a(t) defined in (4.13). (ii) There is another important property of the F-matrix introduced in (4.18). For a given genus Q of conjoined bases of (H) with Rg(t) in (3.11) let Q(t) be a solution of (TV) on [a, oo) C [«00,00). By Theorem 4.3 we know that also the matrix Q(t) := Rg(t) Q(t) Rg(t) solves (TV) on [a, 00). Moreover, let 9(t) and 9(t) be the corresponding matrices from Remark 4.10, that is, 9' = [A(t) + B(t) Q(t)] 9, 9' = [A(t) + B(t) Q(t)] 9, is [a, 00), 9(a) = Rg(a) = 9(a). With the aid of the identities Rg(t) 9(t) = 9(t) and B(t)Rg(t) = B(t) on [a, 00) we obtain the equality e'(t) (4=1} [A(t) + B(t) Q(t)] G(t) = A(t) G(t) + B(t)Rg(t) Q(t) Rg(t) 9(t) = A(t) 9(t) + B(t) Q(t) 9(t) = [A(t) + B(t) Q(t)] 9(t) for every t € [a, 00). Therefore, the matrices 9(t) and 9(t) solve the same equation on [a, 00) and hence, 9(t) = 9(t) for all t € [a, 00) by the last condition in (4.21). Consequently, the matrices Fa(t) and Fa(t) in (4.18) associated with the solutions Q(t) and Q(t), respectively, satisfy the equality Fa(t) = Fa(t) on [a, 00). The representation of the matrix Fa(t) in (4.20) in terms of the fundamental matrix <&a(t) of (4.13) allows to apply the original result in [23, Lemma 2.1, pg. 12] for symmetric solutions Q(t) of (7?.). This yields the following statement, which will be utilized in our further analysis. Proposition 4.12. Let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (3.11) and let Q(t) be a symmetric solution of the Riccati equation (TV) on [a, 00) C [aoo,oo). Moreover, let <&a(t) and Fa(t) be the corresponding matrices in (4.13) and (4.20), respectively. Then an n x n matrix-valued function Q(t) solves (R) on [a, 00) if and only if the constant matrix G := Q(a) — Q(a) is such that the matrix I + Fa(t) G is nonsingular on [a, 00) and Q(t) = Q(t) + -1(t) for every te [a, 00). (4.22) Remark 4.13. Let K be a given n x n matrix and let Q(t) be the solution of the Riccati equation (TV) satisfying Q(a) = K. From Proposition 4.12 it then follows that the matrix Q(t) as a solution of (TV) can be extended to the whole interval [a, 00) if and only if the matrix G := K — Q(a) is such that the matrix I+Fa(t) G is nonsingular for all t € [a, 00). In this case, the solution Q(t) has the representation in (4.22). riccati equations for linear hamiltonian systems 1705 Formula (4.22) allows to derive inequalities between symmetric solutions of the Riccati equation (TV). We note that the statement about the constant rank of Q(i) — Q(t) corresponds to [23, Corollary 2, pg. 13]. Corollary 4.14. Let Q be a genus of conjoined bases of (H) with the corresponding matrix Rg(i) in (3.11) and let Q(t) and Q(i) be symmetric solutions of the Riccati equation (TV) on [a, oo) C [«00,00). Then the quantities rank [Q(t) — Q(t)] and ind [Q(t) — Q(t)] are constant on [a, 00). (4.23) In particular, the inequality Q(t) > Q(t) holds on [a, 00) if and only ifQ(a) > Q(a), and the inequality Q(t) > Q(i) holds on [a, 00) if and only if Q(a) > Q(a). Proof. Let Q(i) and Q(t) be as in the corollary and set G := Q(a) — Q(a). With the aid of formula (4.22) we then obtain that rank [Q(t) — Q(t)] = rankG = rank [Q(a) — Q(a)] on [a, 00). Moreover, the continuity of the matrices Q(t) and Q(t) implies that also the quantity ind [Q(t) — Q(t)] is constant on [a, 00), completing the proof of the statements in (4.23). Finally, the assertions in the second part of the corollary follow immediately from (4.23). □ The next statement extends to an arbitrary genus Q the result in [7, Corollary (iv), pp. 52-53], in which we consider one system (H). Corollary 4.15. Assume (1.1) and let Q be a genus of conjoined bases of (H) with the corresponding matrix Rg(t) in (3.11). Let Q(t) be a symmetric solution of the Riccati equation (TV) on [a, 00) C [«oo,oo) and let Q(i) be a symmetric solution of (TV) satisfying the condition Q(a) > Q(a). Then the matrix Q(t) solves (TV) on the whole interval [a, 00) such that the inequality Q(t) > Q(i) holds for all t € [a, 00). Proof. Let Fa(t) be the matrix in (4.18) associated with Q(t) on [a, 00) and set G := Q(a) — Q(a). We will show that the matrix / + Fa(t) G is nonsingular on [a, 00). Fix t € [a, 00) and let v € Ker [I+Fa(t) G], i.e., the equality v = —Fa(t) Gv holds. Since the matrix G is symmetric and satisfies G > 0 and from Remark 4.11 we know that under the Legendre condition (1.1) the matrix Fa(t) is nonnegative definite, we have that 0 < vTGv = -vTGTFa(t)Gv < 0. Thus, vTGv = 0 and consequently, Gv = 0. Therefore, the vector v = —Fa(i)Gv = 0 and the matrix I + Fa(t)G is nonsingular. Finally, according to Remark 4.13 and Corollary 4.14 this then means that the matrix Q(i) solves the Riccati equation (7?.) on the whole interval [a, 00) and satisfies the inequality Q(i) > Q(t) for every t € [a, 00). □ In the next result we show further properties of the solutions of the Riccati equation (7?.). Namely, we characterize a certain class of the values K of the initial conditions at some point ft, which guarantee that the corresponding solution Q(i) of (TV) with Q(/3) = K exists on the whole interval [5, 00). Another interpretation of the following statement is that any symmetric solution Q(t) of (TV) on [a, 00) C [«oo, 00) satisfying inclusion (1.4) can be decomposed into the product Rg(i) Q(i) Rg(t) for a suitable, in general nonsymmetric, solution Q(t) of (TV). This result can be regarded as a partial converse to Theorem 4.3 and it will be utilized for the classification of solutions of (TV) in Remark 4.20 below. Theorem 4.16. Let Q be a genus of conjoined bases of (H) and let Rg(i) be its corresponding matrix in (3.11). Moreover, let Q(i) be a symmetric solution of (TV) on [a, 00) C [«oo,oo) satisfying condition (1.4). Let ft € [a, 00) and K € R™x™ be 1706 peter šepitka given and consider the solution Q(t) of (TV) with Q((3) = K. Then the following statements are equivalent. (i) The matrix Q(i) solves the Riccati equation (TV) on the whole interval [a, oo) such that Rg(t) Q(i) Rg(i) = Q(t) holds for every t € [a, oo). (ii) The matrix K satisfies the equality Rg(/3) KRg(/3) = Q((3). Proof. First we note that assertion (i) implies (ii) trivially. Therefore, suppose that (ii) holds, i.e., the matrix K satisfies Rg(/3) KRg(/3) = Q(/3). Let $a(i) and Fa(t) be the matrices in (4.13) and (4.18) associated with Q(t) and put E:=I- C(/3) [K - Q(f3)} $a(/3) Fa(f3). (4.24) We observe that from Remark 4.11(i), inclusion (1.4) with symmetric Q(i), and (4.17) at t = f3 we have Fa(f3) = Rg(a)Fa(f3), Rg(f3)Q(f3) { = ] Q(f3) { = ] Q(f3)Rg(f3), Rg(f3)<5>a(f3)Rg(a) { =?) $a(/3) Rg(a). Then the matrix E in (4.24) satisfies Rg(a) E (A=] Rg(a) - Rg(a) $£(/3) [K - Q(f3)} $a(/3) Fa(f3) ( =5) Rg(a) - Rg(a) $£(/3) [Rg(f3) KRg(f3) - Q(f3)} $a(/3) Rg(a) Fa(f3) = Rg(a). (4.26) We will show that the matrix E is nonsingular. Let v € Ker E. This means according to (4.24) that v = C 03) [K -Q(f3)} (13) Fa (f3) v. (4.27) Moreover, from (4.26) it follows that Rg(a)v = Rg(a) Ev = 0. Combining the latter equality together with (4.27) and with the first identity in (4.25) yields v (4=7) C(/3) [K - Q([3)] $a(/3) Fa([3)v = C(/3) [K - Q(/3)] $a(/3) Fa(f3) Rg(a) v = 0, which proves the nonsingularity of E. In particular, formula (4.26) is then equivalent with the equality Rg(a) E~l = Rg(a). Now set G := S"1 C(/3) " Q(/3)] T(f3) [K - Q(f3)} $a(/3) Rg(a) = Rg(a) C(/3) [K - Q(f3)] $a(/3) Rg(a) ( =5) i?s(a) C(/3) [^(/3) KRg(f3) - Q(f3)} $a(/3) i?s(a) = 0. (4.30) Fix t € [a, oo) and let « € Ker [/ + Fa(t) G], i.e., the equality v = —Fa(t) Gv holds. In particular, the vector v € lmFa(t) C ImiJ6(a), by Remark 4.11(i). This means that v = Rg(a)v, which in turn together with the equality Fa(t) Rg(a) = Fa(t) and equation (4.30) yields v = -Fa(t)Gv = -Fa(t) Rg(a) GRg(a) v = 0. Thus, the matrix / + Fa(t) G is nonsingular on [a, oo) and by Remark 4.13 the solution Q*(t) exists on the whole interval [a, oo) such that Q.(t) = Q{t) + C"1 (t) G[I + Fa(t) G}-1 -\t) (4.31) for all t € [a, oo), by (4.22). In particular, it follows for the matrix Q*(f3) that W) (=1} Q(f3) + C_1(/3) G [I+Fa(f3) G]-1 $-1(/3) (=9) Q(f3) + [K — Q(/3)] = K = Q(/3). Therefore, by the uniqueness of solutions of (TV) the matrix Q(t) solves (TV) on the whole interval [a, oo) with Q(t) = Qt(t) for every t € [a, oo). In turn, the matrix Rg(t) Q(t) Rg(t) is also a solution of (TV) on [a, oo), by Theorem 4.3. Finally, since Rg(/3) Q(/3) Rg(/3) = Rg(j3) KRg(/3) = Q(/3), we conclude by using the uniqueness of solutions of (TV) once more that Rg(t) Q(t)Rg(t) = Q(t) for all t € [a, oo). This shows (i) and the proof is complete. □ Remark 4.17. For the completeness we note that the matrix Q(t) in Theorem 4.16 satisfies the formula Q(t) = Q(t) + C"1 (t) G[I + Fa(t) G]"1 ^-\t), t e [a, oo). This follows directly from the equality Q(t) = Qt(t) on [a, oo) and the representation of the matrix Qt(t) in (4.31). We are now ready to formulate the main results of this section (Theorems 4.18 and 4.21), in which we connect the solutions Q(t) of the Riccati equation (TV) on [a, oo) with conjoined bases (X, U) with constant kernel on [a, oo) from the genus Q. These results extend the well known correspondence between the symmetric solutions Q(t) of (R) on [a, oo) and conjoined bases (X, U) of (H) with X(t) invertible on [a, oo), i.e., Q(t) = U(t)X~1(t) on[a,oo) (4.32) to the case of possibly noninvertible X(t) on [a, oo). Theorem 4.18. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (3.11). Moreover, let (X, U) be a conjoined basis of (H) belonging to Q such that (X,U) has constant kernel on a subinterval [a, oo) C [«00,00) and let Q(t) be the corresponding Riccati quotient in (2.7). Then the matrix Q(t) is a symmetric solution of the Riccati equation (TV) on [a, 00) such that the condition in (1.4) holds and the matrices 0(i) and fl(t) defined by Q(t):=X(t), n(t):=U(t)-Q(t)X(t), te[a,oo), (4.33) 1708 peter šepitka solve the initial value problem (4.9)-(4.10) on [a, oo) with (4.11). Proof. Let R(t) be the orthogonal projector in (2.4) associated with (X, U). From Proposition 3.11 we know that R(t) = Rg(t) on [a, oo). By (2.7) the matrix Q(t) then satisfies the equality Q(t) = Rg(t) U(t) X^ (t) for every t € [a, oo). Moreover, using the identities in (1.3), (3.12), and (4.2) we obtain on [a, oo) that Q' = Rg UX^ + Rg U'X^ + Rg U(X^Y {=2) [Rgj AT] UX^ + Rg (CX - ATU) Xt + Rg U(X^)' {3=2) RgATUX] - ATRg UX] + Rg CRg - RgATUX] - Rg UX] (A + BQ) {1=3) -QA-ATQ-QBQ + C. Thus, the matrix Q(t) solves the Riccati equation (TV) on [a, oo) and condition (1.4) holds. Furthermore, according to (3.12) in Proposition 3.11 the matrix 9(i) = X(t) satisfies the first equation in (4.9) on [a, oo) while applying (3.13) and (TV) yields for the matrix fl(t) the equality íl' -Ail = (U -QX)' -A(U -QX) (3=3) [c -(A + AT) Q]X - Q'X - QX' +AQX (w),X3.i2) [c _ (A + AT^ q] x + (a + AT) QX - CX (4.34) on [a, oo). Moreover, by using (4.5), (1.3), and the equalities Rg(t) Q(t) = Q(t) and Rg(t)X(t) = Rg(t) for every t € [a, oo) the last two terms in (4.34) become (A + AT) QX - CX {1=} (A + AT) Rg QX - Rg CRgX {=5) Rg (A + AT) Rg QX - Rg CX = Rg(A + AT) QX - Rg CX = -Rg [C - (A + AT) Q] X (4.35) on [a, oo). By combining formulas (4.34) and (4.35) we obtain that n' - An = (I - Rg) [C - (A + AT) Q]X = (I - Rg) [C - (A + AT) Q] 9 on [a, oo), showing the second equation in (4.9). Finally, from the first identity in (4.33) we have that Im9(a) = Iml(a) = lmRg(a), while the second one together with the last formula in (2.8) give Rg(a) tt(a) = Rg(a) U(a) - Rg(a) Q(a) X(a) = Rg(a) U(a) - Q(a) X(a) = 0. Hence, the inclusion lmfl(a) C Ker Rg(a) holds. 9n the other hand, with the aid of (4.33) and the fact that (X, U) is a conjoined basis one can easily check that Ker9(a) nKerfl(a) = KerX(a) n KerU(a) = {0}, which is equivalent with the equality rank(9T(a), flT(a))T = n. Therefore, the matrices 9(a) and fl(a) satisfy the conditions in (4.11) and the proof is complete. □ Remark 4.19. Let Sa(t) be the matrix in (2.11) associated with the conjoined basis (X, U) on [a, oo) and let Fa(t) be the matrix in (4.18), which corresponds to riccati equations for linear hamiltonian systems 1709 the Riccati quotient Q(t) on [a, oo). The identities in (3.14) and (4.20) then give Sa(t) = f Xt (s) B(s) X^T(s) As ( =4) f Xt (a) ^(s) B(s) C"1 (s) X^T (a) As {1=3) Xt (a) f (s) B(s) C"1 W dsX^(a) {4=0) X1" (a) F« (t) XtT (a) (4.36) for all t € [a, oo), where $a(i) is the fundamental matrix in (4.13). On the other hand, with the aid of the equalities X(a) X^ (a) = Rg(a) = X^T(a)XT(a) and Fa(t) = Rg(a) Fa(t) Rg(a) on [a, oo) expression (4.36) yields the formula Fa(t) = Rg(a)Fa(t) Rg(a) = X(a) X\a) Fa(t) XtT(a) XT(a) {4=6) X(a)Sa(t)XT(a). (4.37) for every on t g [a, oo). Furthermore, let P, Psa(t), and Psaoo be the matrices in (2.4) and (2.12) associated with (X,U). By combining (4.37) with the identities XT(a)XtT(a) = P and Sa(t) P = Sa(t) we get Fa(t)X^T(a) = X(a)Sa(t)P = X(a) Sa(t), which in turn through (2.24) implies that lmFa(t) = ImX(a) Sa(t) {2=} ImX(a) PSa (t) {2=} (A^a,^, (4.38) where the subspace Aa[a,t] is defined in Section 2. Note that equality (4.38) is in a full agreement with the monotonicity of the subspace Ker Fa(t) in Remark 4.11 (i). Moreover, by using the relation in (2.22) we obtain lmFa(t) (4=8) (Acja,*])1- {i=] (A^a.oo))1 on (r„i00,oo), (4.39) where the point r„i00 is defined in (2.23). Remark 4.20. Based on Theorem 4.f8, the result in Theorem 4.f6 enables to determine all the solutions Q(t) of the Riccati equation (7?.) on [a, oo), for which the matrix Rg(t) Q(t) Rg(t) is the (symmetric) Riccati quotient associated with the conjoined basis (X, U) on [a, oo). More precisely, if ft € [a, oo) is a given point and Q(t) is a solution of (TV) defined on a neighborhood of ft, then the matrix Q(t) solves (TV) on the whole interval [a, oo) and the matrix Rg(t) Q(t) Rg(t) is the Riccati quotient associated with (X, U) for every t € [a, oo) if and only if the matrix Rg(/3) Q(/3) Rg(/3) is the Riccati quotient for (X,U) at the point /3. In addition, from Remarks 4.11 (ii) and 4.f9 it follows that the matrix Fa(t) in (4.18), which corresponds to every such a solution Q(t), satisfies formulas (4.36)-(4.39). Theorem 4.21. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (3.11) and let Q(t) be a solution of the Riccati equation (TV) on [a, oo) C [a^, oo) such that the matrix Rg(t) Q(t) Rg(t) is symmetric on [a, oo). Moreover, let (O, Q,) be a solution of (4.9)-(4.11) on [a, oo) and define the matrices X(t):=Q(t), U(t):=Q(t)Q(t) + tt(t), ie[a,oo). (4.40) Then the following statements hold. (i) The pair (X, U) is a conjoined basis of (H) such that (X, U) has a constant kernel on [a, oo) and belongs to the genus Q. (ii) The matrix Rg(t) Q(t) Rg(t) is the Riccati quotient in (2.7) associated with (X,U) on [a, oo), i.e., the equality Rg(t) Q(t) Rg(t) = R(t)U(t) X\t) holds for all t € [a, oo), where R(t) is the corresponding projector in (2.4). 1710 peter šepitka Proof, (i) First we show that the pair (X, U) is a solution of (H) on [a, oo). From Proposition 4.9 we know that the matrix 9(i) has constant kernel on [a, oo) and Im9(i) = lmRg(t) for every t € [a, oo). Thus, the matrix X(t) has constant kernel on [a, oo) and lmX(t) = Rg(t) for all t € [a, oo). And since 9(i) solves the first equation in (4.9) on [a, oo), we have that X'(t) = [A(t) + B(t)Q(t)]X(t), te[a,oo). (4.41) Moreover, by using (1.3), (4.40), and the inclusion in (4.7) we obtain the formula B(t) U(t) (1-3)' (4=0)' {4'7) B(t) Q(t) X(t) + Bit) Rg(t) ^t) = Bit) Qit) X(t) (4.42) for every t € [a, oo). Combining identities (4.41) and (4.42) with (4.3) and the equality Rg(t) X(t) = X(t) for every t € [a, oo) then yields on [a, oo) that (A 41 *| (A A'l) (A 3*1 X'=' AX + BQX [ = ' ARgX + BU = ARgX + BU = AX + BU. (4.43) Next we derive some additional properties of the matrices 9(i) and fi(t), which will simplify our calculations. In particular, by (TV) and the first equation in (4.9) we have on [a, 00) that (QO)' ^i|4-9) (C-QA-aTQ-QBQ)® + Q(A+BQ)®= (C-ATQ)®. (AAA) On the other hand, by (1.3) and the identities Rg(t) Vt(t) = 0 and Rg(t) 6(t) = Q(t) for all t € [a, 00), the second equation in (4.9) reads on [a, 00) as n' {4-9)={1-3) [ARg -At(i- Rg)] £1+ [C - (A + AT) Q]Q - Rg [C - (A + AT) Q] 9 = -ATn +[C- ATQ] O - AQO - RgCRge + Rg(A + AT) Q9 (i=3) _ATn + [c _ aTq^ q_cq + ^r^a + aT)- A]Q9 (i=3) _ATn + [c _ aTq^ e _ [c _ atq] 9. (4.45) Now by using (4.44) and (4.45) we obtain that the matrix U(t) in (4.40) satisfies u' = (Qey + n' = (c - atq)9- ATn + [c- atq]9- \c - atq]e = C®-AT(Q® + n){4=)CX-ATU on [a, oo). (4.46) Hence, equalities (4.43) and (4.46) show that the pair (X, U) solves system (H) on [a, oo). Moreover, the matrix xT(t) u(t) = eT(t) Q(t) e(t) + eT(t) n(t) = eT(t) pg(t) Q(t) pg(t) e(t) is symmetric and the subspace KerX(t) n Ker[7(t) = Ker9(t) n KerSl(t) = {0} for every t € [a, oo), both by Proposition 4.9. Therefore, the solution (X,U) is a conjoined basis with constant kernel on [a, oo). And since the equality lmX(t) = lmRg(t) holds for every t € [a, oo), we conclude that (X, U) belongs to the genus Q. For the proof of part (ii) we note that the orthogonal projector R(t) in (2.4) associated with (X,U) satisfies R(t) = Rg(t) for all t € [a, oo). In particular, by using the identities 9(t) (t) = Rg(t) and Rg(t) il(t) = 0 on [a, oo) we obtain that R(t) U(t) X\t) {4=0) Rg(t) Q(t) Q(t) &(t) + Rg(t) il(t) &(t) = Rg(t) Q(t) Rg(t) for every t € [a, oo). Thus, the matrix Rg(t) Q(t) Rg(t) is the Riccati quotient in (2.7) associated with (X, U) on [a, oo) and the proof is complete. □ riccati equations for linear hamiltonian systems 1711 Remark 4.22. We note that the matrices Fa(t) and Sa(t) in (4.18) and (2.11) associated with the matrix Q(t) and the conjoined basis (X, U) in Theorem 4.21, respectively, satisfy the identities in (4.36) and (4.37). This follows directly from Theorem 4.18 and Remark 4.11 (ii). Remark 4.23. Let Q be a genus of conjoined basis of (H) with the associated matrix Rg(t) in (3.11) and let [a, oo) C [«00,00) be a given interval. The results in Theorems 4.18 and 4.21 provide a correspondence between the set of all conjoined basis (X, U) of (H) with constant kernel on [a, 00), which belong to the genus Q, and the set of all symmetric solutions Q(t) of the Riccati equation (TV) on [a, 00) satisfying condition (1.4). More precisely, for every such a conjoined basis (X,U) its Riccati quotient Q(t) in (2.7) is a symmetric solution of (TV) on [a, 00) with ImQ(£) C lmRg(t) for all t € [a, 00), as we claim in Theorem 4.18. Conversely, if Q(t) is a symmetric solution of (TV) on [a, 00) such that ImQ(£) C lmRg(t) for every t € [a, 00), then there exists a conjoined basis (X, U) of (H) from the genus Q with constant kernel on [a, 00) such that the matrix Q(t) is its corresponding Riccati quotient in (2.7), by Theorem 4.21, and the equality Rg(t) Q(t) Rg(t) = Q(t) on [a, 00). In addition, every such a conjoined basis (X,U) has the form of (4.40) for some solution (0,f2) of (4.9)-(4.11) on [a, 00). Finally, the observations in Remark 4.8 then imply that the conjoined basis (X, U) is uniquely determined up to a right nonsingular multiple by the genus Q and the matrix Q(t). Remark 4.24. The representation of the solution Q(t) of (R) in (4.32) corresponds to the results in Theorems 4.18 and 4.21 with the maximal genus Q = £/max- In this case Rg(t) = I, and (1.3) yields that the Riccati equations (R) and (TV) coincide. 5. Inequalities for Riccati quotients in given genus. In this section we derive a mutual representation of the Riccati quotients corresponding to conjoined bases of (H) from a given genus Q (Theorem 5.3). This representation is then utilized for obtaining inequalities between two Riccati quotients (Corollary 5.5). The results presented in this section essentially generalize the discussion in [7, pg. 54] to possibly uncontrollable systems (H). First we prove an auxiliary property of the image of the matrix Fa(t) in (4.18). Lemma 5.1. Assume (1.1). Let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (3.11) and let Q(t) be a solution of the Riccati equation (TV) on [a, 00) C [«00,00) such that the matrix Rg(t) Q(t) Rg(t) is symmetric on [a, 00). Moreover, let Fa(t) be the matrix in (4.18), which corresponds to Q(t), and RAoa(t) be the orthogonal projector defined in (3.7). Then ImFtt(t) C Im RAoa(a) for every t € [a, 00). (5-1) Proof. Let (X, U) be a conjoined basis of (H) from the genus Q, which corresponds to the matrix Q(t) through Theorem 4.21. It follows that (X, U) has constant kernel on [a, 00) and the matrix X(t) satisfies the equality lmX(t) = lmRg(t) for every t € [a, 00). Moreover, the matrix Rg(t) Q(t) Rg(t) is the Riccati quotient in (2.7) associated with (X, U) on [a, 00), by Theorem 4.21(h). Let Sa(t) be the 5-matrix in (2.11) corresponding to the conjoined basis (X, U) on [a, 00). From Remark 4.11(h) we know that Fa(t) is the F-matrix in (4.18) associated with Rg(t) Q(t) Rg(t). Thus, by combining (3.8) and (4.39) we obtain the identity (4-39) ±_ (3 8) lmFa(t) = (Aa[a, 00]) = lmRAoa(a) on (raj00,00), (5-2) 1712 peter šepitka where the point raj oo is defined in (2.23). And since by Remark 4.11 (i) the subspace ImFtt(t) is nondecreasing on [a, oo), the inclusion in (5.1) now immediately follows from (5.2). The proof is complete. □ Remark 5.2. (i) Let Sa(t) and Psaoo be the matrices in (2.11) and (2.12) which correspond to the conjoined basis (X,U) on [a, oo) C [«00,00). From (2.25) and (5.2) it follows that lm,Sa(t) = lmPSaOC, lmFa(t) = lmRAoo(a), (5.3) rank Sa(t) = rankPga00 = n — d[a, 00) = ranki?Aoo(a) = rankFa(t) (5.4) on (raj00,00). Moreover, the matrices Sa(t) and Fa(t) satisfies the identities SUt) = Psaoo XT(a) Fl(t) X(a) PSa00, Fl(t) = iW«) X^T{a) Slit) Xt(a) RAoa(a) for every t € (Ta,oo,oo), which can verify by direct computation with the aid of (3.9)-(3.10), (4.36), (4.37), and (5.3). (ii) Furthermore, upon taking t —> 00 in (5.5) we obtain the formulas Ta = PSaoo XTia) Da X(a) PSaoo, Da = RAoo(a)X^T(a)TaX^(a)RAoo(a), where the matrices Ta and Da are defined in (2.13) and (4.19), respectively. The equalities in (5.6) then yield that lvnDa C 1m RAao(a) and rankTa = rankDa. In particular, combining the last formula with Remark 2.8(h) and the fact that d[a, 00) = rfoo implies that rankl)^ is constant with respect to /3 € [a, 00). In the following main result of this section we present a representation of two Riccati quotients corresponding to two conjoined bases of (H) from a genus Q. This result will be utilized in the classification of all distinguished solutions of (TV) at infinity in Section 7. When Q = Qma.K is the maximal genus (in particular, when system (H) is controllable), this representation coincides with the statement in Proposition 4.12. We note that for a given genus Q we now compare those solutions Q(t) and Q(t) of (TV), which are Riccati quotients according to their definition in (2.7). However, the Riccati equation (TV) may also have other solutions, which are not of this particular form. Theorem 5.3. Let Q be a genus of conjoined bases of (H) and let (X, U) and (X,U) be two conjoined bases of (H) with constant kernel on [a, 00) C [«00,00) belonging to Q. Moreover, let P and Sa(t) be the matrices in (2.5) and (2.11) associated with (X,U). Suppose that (X,U) is expressed in terms of (X,U) via matrices M and N as in Proposition 2.10. Then the Riccati quotients Q(t) and Q(t) in (2.7) corresponding to (X,U) and (X,U), respectively, satisfy Q(t) = Q(t)+X^T(t)N[PM + Sa(t)N]^X^(t), te[a,oo). (5.7) Proof. Let Rg(t) be the orthogonal projector in (3.11) and let R(t) and R(t) be the matrices in (2.4), which correspond to the conjoined bases (X, U) and (X, U), respectively. According to Proposition 3.11 and the second identity in (2.4) we then have the equalities X(t) X\t) {=4) R(t) = Rg(t) = R(t) {=4) X(t) X\t) (5.8) riccati equations for linear hamiltonian systems 1713 for all t € [a, oo). Moreover, the symmetry of the matrices Q(t), Q(t), R(t), and R(t) on [a, oo), the fact that the matrix TV is the Wronskian of (X, U) and (X, U) by Remark 2.11(i), and the equations in (5.8) imply that (we omit the argument t) Q-Q ( = ] RUX^-R UX* (=8) R UX^ - X^TUTR (2=4) XWxTvxi _ XfT UTXX\ = X^T(XTU - UTX) X1" = X^iVX1" (5.9) on [a, oo). Finally, inserting the expression for the matrix X (t) in (2.37) into the equality in (5.9) yields formula (5.7) on [a, oo) and the proof is complete. □ Remark 5.4. By substituting the matrix X^ (t) instead of X (t) in (5.9) we get another formula for the difference Q(t) — Q(t). Namely, inserting the second identity in (2.37) into (5.9) and using the equality PX = X and the symmetry of Sa(t) on [a, oo) yields the formula Q(t) - Q(t) = XtT(t) [PM + Sa(t) N]TNX\t) = X]T(t) [MTX + XTSa(t) X] X](t) (5.10) for every t € [a, oo). In addition, if P is the projector in (2.5) associated with the conjoined basis (X, U), then by the identities XTXT = P = X^X on [a, oo), XP = X, and MTX = XTM formula (5.10) implies (suppressing the argument t) XT[Q -Q]X (5=0) XTX^T[MTX + XTSa X] X^X = P[MTX + XTSaX]P = MTX + XTSaX (5.11) on [a, oo). Moreover, from (5.10) and (5.11) it immediately follows that rank [Q(t) - Q(t)} = rank [MTX + XTSa(t) X], 1 ind[Q(t) -Q(t)} =md[MTX + XTSa(t)X] J for every t € [a, oo). In particular, since Sa(a) = 0, by evaluating (5.12) at t = a we obtain the equalities rank [ Q(a) - Q(a)} = rank MTX, ind [ Q(a) - Q(a)} = ind MTX. (5.13) Formula (5.7) in Theorem 5.3 yields the following inequalities between two Riccati quotients associated with two conjoined bases from the genus Q. Corollary 5.5. With the assumptions and notation of Theorem 5.3, the Riccati quotients Q(t) and Q(t) satisfy the formulas rank[Q(t) - Q(t)] = rank TV, ind [ Q(t) - Q(t)} = ind iVM-1 (5.14) on [a, oo). Moreover, the following statements hold. (i) The inequality Q(t) > Q(t) holds for all t € [a, oo) if and only if XM~l > 0. (ii) The inequality Q(i) < Q(t) holds for all t € [a, oo) if and only if XM~l < 0. (iii) The inequality Q(t) > Q(t), resp. Q(t) < Q(t), holds on the subspacelm Rg(t) for all t € [a, oo) if and only if the inequality XM~l > 0, resp. XM~l < 0, holds on Im P. 1714 peter šepitka Proof. From Theorem 4.18 we know that the matrices Q(i) and Q(i) are symmetric solutions of (TV) on [a, oo). Therefore, the quantities rank[Q(i) — Q(t)] and ind [Q(t) — Q(t)] are constant on [a, oo), by Corollary 4.14. According to (2.31) the matrix M is nonsingular and the matrix MTN is symmetric. Thus, the matrix MT~1MTNM~1 = NM'1 is symmetric and rank MTN = rank NM'1 = rank N, ind MTN = ind NM'1. (5.15) The formulas in (5.14) now follow from (5.13) and (5.15). Furthermore, assertions (i) and (ii) are direct consequences of the equalities in (5.14). For the proof of statement (iii) we note that the matrix Q(t) —Q(t) satisfies the inclusion Im [Q(t) — Q(t)} C lmRg(t) for all t e [a, oo) by Theorem 4.18, while the matrix NM-1 satisfies the inclusion ImTVM-1 C Im P by Proposition 2.10. Moreover, the equality ranki?s(t) = rankP =: r holds on [a, oo), by (2.6) and (5.8). Now if Q(t) > Q(t) on lvnRg(i) for every t € [a, oo), then we have that rank [Q(i) — Q(t)] = r and ind [Q(t) — Q(t)] = 0 on [a, oo). Consequently, by using (5.14), we obtain the equalities rankTVM-1 = rank TV = r and indTVM-1 = 0. This means that the matrix NM~l satisfies NM~l > 0 on ImP. Conversely, if the inequality NM~l > 0 holds on ImP, then rank TV = rankTVM-1 = r and indTVM-1 = 0. Therefore, rank[Q(t) - Q(t)] = r and ind [Q(t) - Q(t)] = 0 on t € [a,oo), by (5.14). This then shows that Q(i) > Q(t) on lmRg(t) for all t € [a, oo). Finally, the opposite inequalities can be proven in a similar way. The proof is complete. □ Remark 5.6. As a completion of Corollary 5.5 we note that the matrices Q(t) and Q(i) are equal on the whole interval [a, oo) if and only if JV = 0. In this case, according to (2.30), the conjoined bases (X, U) and (X, U) satisfy X(t) = X(t) M and U(i) = U(t) M on [a, oo), and hence on [a, oo) by the uniqueness of solutions of (H). This result is in full agreement with the last part of Remark 4.23. 6. Implicit Riccati matrix differential equation. In this section we study solution spaces of the implicit Riccati equations (1.6) and Re(t) [Q' + QA(t) + AT(t) Q + QB(t) Q - C(t)] Rg(t) = 0 (6.1) on [a, oo) C [a^, oo). These implicit Riccati equations were used in [13, Section 6] in several criteria characterizing the nonnegativity and positivity of the associated quadratic functional. The main contributions of this section (Theorem 6.3 and Corollary 6.4) show that under certain assumption we can transfer the problem of solving the implicit Riccati equations (6.1) and (1.6) into a problem of solving the explicit Riccati equation (7?.). In the first result we prove that the two implicit Riccati equations (6.1) and (1.6) are equivalent in terms of their solutions spaces. Lemma 6.1. Let Q be a genus of conjoined bases of (H) and let Rg(t) be the orthogonal projector in (3.11). The sets of solutions of equations (1.6) and (6.1) coincide, i.e., a matrix Q(i) solves (1.6) on a subinterval [a, oo) C [«00,00) if and only if Q(i) solves (6.1) on [a, 00). Proof. Let Q and Rg(t) be as in the lemma and fix a € [«00, 00] • Moreover, let Q(t) be an n x n piecewise continuously differentiable matrix-valued function on [a, 00) riccati equations for linear hamiltonian systems 1715 and define the functions (we omit the argument t) E1 := Rg [Q' + QA + ATQ + QBQ - C] Rg, £2 := Rg [Q' + QA + ATQ + QBQ -C]Rg, (6.2) on [a, oo). By using (1.3) and (4.3) together with the identity [Rg(t)]'2 = Rg(t) for every t € [a, oo) we then obtain (suppressing the argument t) Si { = ] Rs Q'Rs + RgQ ARg + RgATQ Rg + Rg QB QRg - RgCRg (i.ay4.3) R^ QlR^ + RsQ ARg + RgArQ Rg + RgQB QRg - Rg CRg { = ] £2 on [a, oo), which proves directly the statement of the lemma. □ Remark 6.2. It is easy to see that for a given orthogonal projector Rg(t) in (3.11) any matrix Q(t), which solves the Riccati equation (TV) on some subinterval [a, oo) C [«00,00), satisfies also the implicit Riccati equation (6.1) on [a, oo). Following the above remark, we now establish the opposite relation between the solutions of the implicit Riccati equation (6.1) and the Riccati equation (TV). Theorem 6.3. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (3.11). Let Q(t) be a solution of the implicit Riccati equation (6.1) on [a, oo) C [«00,00). Then the matrix Rg(t) Q(t) Rg(t) solves (TV) on [a, 00). Proof. Let Rg(t) and Q(t) be as in the theorem. With the aid of (4.1), (4.2), (6.1), and the equalities (suppressing the argument t) Rg CRg = C and B = Rg BRg on [a, 00) we get (Rg QRg)' = Rg QRg + Rg Q'Rg + Rg QRg (4.iy4.2) /(> ^ + ^ (yl{ +RgQ /» (=1) C ~ (Rg QRg) A - AT(Rg QRg) - (Rg QRg) B (Rg QRg) on [a, 00). Hence, the matrix Rg(t) Q(t) Rg(t) solves (TV) on [a, 00). □ The results in Theorem 6.3 and Lemma 6.1 yield the following. Corollary 6.4. Let Q be a genus of conjoined bases of (H) and let Rg(t) be the corresponding orthogonal projector in (3.11). Moreover, let Q(t) be a symmetric matrix defined on [a, 00) C [«00,00) such that condition (1.4) holds. Then the following statements are equivalent. (i) The matrix Q(t) solves the Riccati equation (TV) on [a, 00). (ii) The matrix Q(t) solves the implicit Riccati equation (6.1) on [a, 00). (iii) The matrix Q(t) solves the implicit Riccati equation (1.6) on [a, 00). Proof. The implication (i) => (ii) follows by Remark 6.2. The equivalence of the assertions in (ii) and (iii) is a direct consequence of Lemma 6.1. Now assume (ii), i.e., suppose that the matrix Q(t) is a solution of (6.1) on [a, 00). The result of Theorem 6.3 and the identities Rg(t)Q(t)Rg(t) {1=4) Q(t) Rg(t) = QT(t) Rg(t) = [Rg(t)Q(t)f {1=4) Q(t) for t € [a, 00) then imply that Q(t) solves (TV) on [a, 00), showing (i). □ 1716 peter šepitka 7. Distinguished solutions at infinity. In this section we study, for a given genus Q, symmetric solutions of the Riccati equation (TV), which correspond to principal solutions of (H) at infinity belonging to the genus Q. This correspondence is based on the results in Theorems 4.18 and 4.21 and in Remark 4.20. We introduce the notion of a distinguished solution of (TV) at infinity (Definition 7.1) and prove its main properties. In particular, we establish the results about distinguished solutions of (TV) at infinity regarding their relationship to principal solutions at infinity (Theorems 7.4 and 7.5) and to the nonoscillation of system (H) at infinity (Theorem 7.8), their interval of existence (Theorem 7.13), their mutual classification within the genus Q (Theorem 7.15), and their minimality in a suitable sense (Theorems 7.16 and 7.18). It may be surprising that these results comply with the known theory of distinguished solutions of the Riccati equation (R) for a controllable system (H) only partially. In many aspects the presented theory for general uncontrollable system (H) is substantially different. This is related to the nature of the problem, since for each genus Q of conjoined bases of (H) there is a different Riccati equation (TV), but even within one genus Q there may be many distinguished solutions of (TV) at infinity. We discuss these issues in Remark 7.25 at the end of this section. We note that the true uniqueness and minimality of the distinguished solution of (7?.) at infinity is satisfied only in the minimal genus Qm-ln (see Theorem 7.23). The following definition extends the notion of a distinguished solution (also called a principal solution) of (R) at infinity for a controllable system (H) in [7, pg. 53]. Definition 7.1. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (3.11). A symmetric solution Q(t) of the Riccati equation (7?.) is said to be a distinguished solution at infinity if the matrix Q(t) is defined on an interval [a, oo) C [0^,00) and its corresponding matrix Fa(t) in (4.18) satisfies Fl(t) -> 0 as t -> 00. The notion in Definition 7.1 also extends the distinguished solution of (R) introduced by W. T. Reid in [21, Section IV] and [23, Section 2.7], which in our context corresponds to the maximal genus Q = £/max (for which Rg(t) = I). Remark 7.2. When it is clear from the context, we will often drop the term "at infinity" in the terminology in Definition 7.1. We also remark that a distinguished solution of the Riccati equation (TV) associated with the genus Q is also defined by the property Da = 0 with the matrix Da in (4.19) corresponding to Fa(t). In the next auxiliary statement we show that the property of being a distinguished solution of (7?.) is invariant under the multiplication by the orthogonal projector Rg(t). This property will be utilized in the proofs of the subsequent main results. Lemma 7.3. Let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (3.11). Let Q(t) be a symmetric solution of the Riccati equation (TV) on the interval [a, 00) C [0^,00). Then Q(t) is a distinguished solution of (TV) at infinity with respect to [a, 00) if and only if the matrix Rg(t) Q(t) Rg(t) is a distinguished solution of (TV) at infinity with respect to [a, 00). Proof. From Theorem 4.3 we know that the matrix Rg(t) Q(t) Rg(t) solves (TV) on [a, 00). And since by Remark 4.11(h) the matrices Q(t) and Rg(t) Q(t) Rg(t) have the same F-matrices in (4.18) with respect to the interval [a, 00), the statement follows directly from Definition 7.1. □ riccati equations for linear hamiltonian systems 1717 The following two results show that in the context of Theorems 4.18 and 4.21 the distinguished solutions of (TV) correspond to the principal solutions of (H) at infinity from the genus Q. Theorem 7.4. Let Q be a genus of conjoined bases of (H) and Rg(t) be the projector in (3.11). Moreover, let Q(t) be a distinguished solution of (TV) at infinity with respect to the interval [a, oo) C [ttoo, oo). Then every conjoined basis (A, U) of (H), which is associated with Q(t) on [a, oo) via Theorem 4-21, is a principal solution of (H) at infinity with respect to [a, oo) belonging to the genus Q. Proof. Let Rg(t) and Q(t) be as in the theorem. According to Remark 7.2 the matrix Da in (4.19) corresponding to Q(t) satisfies Da = 0. Let (A, U) be a conjoined basis of (H), which is associated with the matrix Q(t) on [a, oo) via Theorem 4.21. Then (A, U) belongs to the genus Q such that (A, U) has constant kernel on [a, oo). Moreover, if Ta is the T-matrix in (2.13) associated with (A, U) through the matrix Sa in (2.11), then we have rankTa = r&nkDa = 0, by Remark 5.2(h). Hence, Ta = 0 and (A, U) is a principal solution at infinity. □ Theorem 7.5. Let (A, U) be a principal solution of (H) at infinity with respect to the interval [a, oo), which belongs to a genus Q. Moreover, let Q(t) be the Riccati quotient in (2.7) associated with (X,U) on [a, oo). Then Q(t) is a distinguished solution of the Riccati equation (TV) at infinity with respect to [a, oo). Proof. By using Proposition 3.2(f) we have the equality d[a, oo) = doo, which means that [a, oo) C [«00,00), by (2.19). Moreover, the matrix Ta in (2.13) associated with (A, U) satisfies Ta = 0. From Theorem 4.18 it follows that the matrix Q(t) is a symmetric solution of the Riccati equation (TV) on [a, 00). Finally, if Da is the matrix in (4.19), which corresponds to Q through its F-matrix Fa(t) in (4.18), then r&nkDa = rankTa = 0, by Remark 5.2(h). Thus, Da = 0 and Q(t) is a distinguished solution at infinity, by Remark 7.2. □ Remark 7.6. We note that according to Theorem 4.18 the distinguished solution Q(t) at infinity in Theorem 7.5 satisfies the additional property (1.4), i.e., the inclusion ImQ(i) C Rg(t) for all t € [a, 00). In particular, the latter relation together with the symmetry of the matrix Q(t) on [a, 00) yields the identity Q(t) = Rg(t) Q(t) Rg(t) for every t € [a, 00). Moreover, from Lemma 7.3 it follows that every symmetric solution Q(t) of (TV) on [a, 00), for which the matrix Rg(t) Q(t) Rg(t) is the Riccati quotient in (2.7) associated with (A, U), is also a distinguished solution at infinity with respect to [a, 00). In general, however, such a matrix Q(t) does not need to satisfy the inclusion in (1.4). From Theorems 7.4 and 7.5 it follows that the property of the existence of a principal solution of (H) at infinity in the genus Q, as stated in [29, Theorem 7.12], transfers naturally to the existence of a distinguished solution at infinity of the associated Riccati equation (TV). Corollary 7.7. Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (3.11). Then there exists a principal solution of (H) at infinity belonging to the genus Q if and only if there exists a distinguished solution of the Riccati equation (TV) at infinity. In this case, the set of all Riccati quotients in (2.7), which correspond to the principal solutions (A, U) of (H) at infinity from the 1718 peter šepitka genus Q, coincides with the set of all matrices Rg QRg, where Q is a distinguished solution of (TV) at infinity. Proof. The statement follows directly from Theorems 7.4 and 7.5 and from Remark 7.6. □ In the following result we characterize the nonoscillation of system (H) in terms of the existence of a distinguished solution of the Riccati equation (TV) in a given (or every) genus Q. This corresponds to [29, Theorems 7.6 and 7.12] regarding the principal solutions of (H) at infinity. Theorem 7.8. Assume (1.1). Then the following statements are equivalent. (i) System (H) is nonoscillatory. (ii) There exists a distinguished solution of equation (TV) for some genus Q. (iii) There exists a distinguished solution of equation (TV) for every genus Q. The proof of Theorem 7.8 is displayed below after the following two remarks. Remark 7.9. The result in Theorem 7.8 justifies the development of the theory of genera of conjoined bases for possibly oscillatory system (H). Of course, assuming that system (H) is nonoscillatory, then it is sufficient to use the theory of genera of conjoined bases from [29, Section 6] and [31, Section 4] for the construction of distinguished solutions of the Riccati equation (7?.) for a genus Q. It is the converse to this implication, which requires a more general approach, since in this case we need to define the coefficients of equation (TV) without the assumption of nonoscillation of system (H). This natural requirement was the initial motivation for the study presented in [27]. Remark 7.10. We note that the result in Theorem 7.8 remains valid also with the additional condition (1.4) for solutions Q(t) of (TV) in parts (ii) and (iii). More precisely, system (H) is nonoscillatory if and only if there exists a distinguished solution of (TV) at infinity for some (and hence for every) genus Q, which satisfies condition (1.4) for all sufficiently large t € [«00,00). This observation follows directly from Lemma 7.3 and Theorem 7.8. Proof of Theorem 7.8. If (H) is nonoscillatory, then by Remark 3.13 for any genus Q of conjoined bases of (H) there exists a principal solution of (H) at infinity belonging to Q. In turn, there exists a distinguished solution of the Riccati equation (TV) at infinity for every genus Q, by Corollary 7.7. Moreover, assertion (iii) implies (ii) trivially. Finally, by using Corollary 7.7 once more, assertion (ii), that is, the existence of a distinguished solution of (TV) at infinity for some genus Q, means that there exists a principal solution of (H) at infinity, which belongs to Q. Since every principal solution is a nonoscillatory conjoined basis, system (H) is nonoscillatory, by Proposition 2.1. This shows the validity of (i) and completes the proof. □ The next two results deal with the interval of existence of distinguished solutions of (TV). In particular, we determine the maximal interval of existence for each particular distinguished solution of (TV). Moreover, we show that this maximal interval is the same for all distinguished solutions of (TV) as well as for all genera Q. Theorem 7.11. Assume (1.1) and let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (3.11). Moreover, let Q(t) be a distinguished solution of the Riccati equation (TV) at infinity with respect to the interval [a, 00) C [«00,00). riccati equations for linear hamiltonian systems 1719 Then the matrix Q(t) is a distinguished solution of (TV) at infinity also with respect to the interval [/3, oo) for every /3 > a. Proof. Let (X, U) be a conjoined basis of (H) corresponding to Q(t) on [a, oo) via Theorem 4.2f. In particular, the matrix Rg(t) Q(t) Rg(t) is the Riccati quotient in (2.7) associated with (X,U) on [a, oo). Moreover, from Theorem 7.4 we know that (X,U) is a principal solution of (H) at infinity with respect to [a, oo), which belongs to the genus Q. Fix now /3 > a. Then (X, U) is a principal solution of (H) at infinity with respect to [/3, oo), by Proposition 3.2(i). Consequently, the matrix Rg(t) Q(t) Rg(t) is a distinguished solution of (7?.) at infinity with respect to [/3, oo), by Theorem 7.5. Finally, by using Lemma 7.3 we conclude that also the matrix Q(t) is a distinguished solution of (7?.) at infinity with respect to the interval [/3, oo). □ Remark 7.12. For a given distinguished solution Q(t) of the Riccati equation (TV) at infinity we define the point Uq € [ctoo, oo) by oiq := inf {a € [a, oo), Q(t) is a distinguished solution of (TV) with respect to [a, oo)}. The result in Theorem 7.11 then implies that Q(t) is a distinguished solution of (TV) with respect to [a, oo) for every a € (ag,oo). In fact, the set (ag,oo) is the maximal open interval on which the matrix Q(t) exists as a solution of (TV). Indeed, if the matrix Q(t) solves the equation (TV) on the interval [a, oo) C [«00,00), then according to Remark 5.2(h) the corresponding matrix Da in (4.f9) satisfies Da = 0, because d[a, 00) = doo, by (2.19). Thus, Q(t) is a distinguished solution of (TV) with respect to the interval [a, 00), by Remark 7.2. Theorem 7.13. Assume that (f.f) holds and system (H) is nonosdilatory with "mm defined in (3.f). Let Q be a genus of conjoined bases of (H) and let Rg(t) be its corresponding orthogonal projector in (3.11). Moreover, let Q(t) be a distinguished solution of the Riccati equation (TV) at infinity with Uq defined in (7.1). Then the equality Uq = am;n holds. Proof. Let a € [«00,00) be such that the matrix Q(t) is a distinguished solution of (TV) with respect to the interval [a, 00). Let (X, U) be a conjoined basis of (H) at infinity with respect to [a, 00), which is associated with Q(t) via Theorem 4.2L Then (X, U) is a principal solution of (H) at infinity with respect to the interval [a, 00), by Theorem 7.4. Moreover, from Theorem 3.4 we know that (X, U) is a principal solution with respect to the maximal open interval (am;n, 00). Thus, we have the inequality am;n < a. And since a € [«00,00) was chosen arbitrarily with regard to Q(t), we obtain that Uq < am;n, by (7.f). Now we show that the last inequality is implemented as the equality. Suppose that Uq < am-ln. According to (7.f) there exists ft € (aQ,amin) such that Q(t) is a distinguished solution of (TV) at infinity with respect to the interval [/3, 00). In turn, the conjoined basis (X, U) in Theorem 4.2f applied to Q(t) := Q(t) on [/3, 00) is a principal solution of (H) at infinity with respect to [/3, 00), by Theorem 7.4. Applying formula (3.3) and Theorem 3.4 with (X, U) := (X, U) then yields the inequality ft > am-ln, which is a contradiction. Therefore, oiq = am-ln holds and the proof is complete. □ 1720 peter šepitka Remark 7.14. Given a genus Q of conjoined bases of (H) with the matrix Rg(t) in (3.11), from Theorem 7.13 it follows that any distinguished solution Q(t) of (TV) is defined on the maximal interval (am;n,oo) and the corresponding matrix Fa(t) in (4.18) satisfies F^(t) —> 0 as t —> oo for every a > am;n. In the following result we present a mutual classification of all distinguished solutions of the Riccati equation (TV). This classification is formulated in terms of the initial values of the involved distinguished solutions at some point a from the maximal interval (am;n,oo). Theorem 7.15. Assume that (1.1) holds and system (H) is nonosdilatory with "mm and RAao(i) defined in (3.1) and (3.7), respectively. Let Q be a genus of conjoined bases of (H) and let Rg(i) be the matrix in (3.11). Moreover, let Q(t) be a distinguished solution of the Riccati equation (TV) at infinity. Then a symmetric solution Q(i) of (TV) defined on a neighborhood of some point a € (amm, oo) is a distinguished solution at infinity if and only if RAoa(a) Q(a) RAoa(a) = RAoa(a) Q(a) RAoa(a). (7.2) Proof. Fix a € (am;n,oo) and let Q(t) be as in the theorem. From Definition 7.1 and Remark 7.14 we know that Q(i) is a distinguished solution of (TV) at infinity with respect to the interval [a, oo). Moreover, let (X, U) be a conjoined basis of (H), which corresponds to Q(i) on [a, oo) via Theorem 4.21. Then (X, U) is a principal solution of (H) at infinity with respect to [a, oo) belonging to the genus Q, by Theorem 7.4. Let Sa(t) be the S-matrix in (2.11) associated with (X, U). Suppose that Q(t) is a distinguished solution of (TV) at infinity. Thus, Q(i) is a distinguished solution of (TV) with respect to [a, oo) and the corresponding conjoined basis (X, U) of (H) in Theorem 4.21 is a principal solution with respect to [a, oo), which belongs to the genus Q. From Proposition 3.14 it then follows that there exist matrices M,X € K"x™ such that X(a) = X(a)M, U(a) = U(a) M + XtT(a) N, (7.3) M is nonsingular, MTX = XTM, ImX C ImP, PSoo NM^P^ = 0, (7.4) where P and Pg^ are the matrices in (2.5), (2.12), and (3.2) associated with the functions X(t) and Sa on (amin, oo), respectively. In particular, the matrices M and TV in (7.3) - (7.4) represent the conjoined basis (X, U) in terms of (X, U) on [a, oo) in Proposition 2.10 and Remark 2.11. Moreover, the matrices Rg(t) Q(t) Rg(t) and Rg(i) Q(i) Rg(t) are the Riccati quotients in (2.7) associated with (X, U) and (X, U) on [a, oo), by Theorem 4.21(h). Consequently, according to (5.7) in Theorem 5.3 with (X, U) := (X, U), (X, U) := (X, U), Q := Rg QRg, Q := Rg QRg, Sa := Sa, M := M, X := X, and by using (2.36) and Sa(a) = 0 we obtain the identity Rg(a) Q(a) Rg(a) = Rg(a) Q(a) Rg(a) + X]T(a) NM^X^a). (7.5) In order to simplify the notation we set Z := RAoo(a)Q(a) RAoo(a), Z := RAoo(a) Q(a) RAoo(a). (7.6) riccati equations for linear hamiltonian systems 1721 Since we have RAao(a) Rg(a) = RAao(a) = Rg(a) RAao(a) and X^(a) RAao(a) = Pg^ X^(a) RAoo(a) by (3.10), formula (7.5) and the last condition in (7.4) imply Z <' = ') RAoa(a) Q(a) RAoa(a) = RAoa(a) Rg(a) Q(a) Rg(a) RAoa(a) {7=5) RAoo(a) [Rg(a)Q(a)Rg(a) + X]T(a) NM^X^a)] RAoo(a) = RAoo(a) Q(a) RAoo(a) + RAoo(a) X]T (a) NM^X^a) RAoo(a) {7=6) z + RAoa(a)X^(a) PSoo NM~lPSoo X\a) RAoa(a) = Z. Thus, with respect to (7.6) the matrix Q(a) satisfies equality (7.2). Conversely, assume that Q(t) is a symmetric solution of (7?.) defined on a neighborhood of a such that the condition in (7.2) holds. We will use the notation in (7.6). Let Fa(t) be the F-matrix corresponding to Q(t) in (4.18) and set G := Q(a) — Q(a). Then we have RAao(a) GRAao(a) = 0, by (7.2). Next we will show that the matrix I + Fa(t) G is nonsingular on [a, oo). Fix t € [a, oo) and let v € K™ be such that [/ + Fa(t) G] v = 0, that is, v = —Fa(t) Gv. The last equality together with (5.1) imply that v € lmFa(t) C ImRAoa(a). Hence, v = RAao(a)v and since Fa(t) = RAao(a) Fa(t) = Fa(t) RAao(a) by the symmetry of Fa(t) and RAao(a), we get v = —Fa(t)Gv = — Fa(t) RAao(a) GRAao(a) v = 0. Therefore, the matrix I + Fa(t) G is nonsingular for every t € [a, oo). Consequently, according to Remark 4.13 with Q := Q, Q := Q, and G := G, we conclude that the symmetric matrix Q(t) solve equation (TV) on the whole interval [a, oo). Let (X, U) be a conjoined basis of (H) associated with Q(i) on [a, oo) via Theorem 4.21. Then (X, U) has constant kernel on [a, oo) and belongs to the genus Q. Therefore, the identities ImX(t) = lmRg(t) = ImX(t) hold for all t € [a, oo), by Remark 3.13. Hence, from Proposition 2.10 and Remark 2.11(i) with (X, U) := (X, U) and (X, U) := (X, U) it then follows that there exists matrices M,N € R™x™ such that the formulas in (7.3) and the first three conditions in (7.4) hold. Similarly as in the first part of the proof, the matrix Rg(i) Q(i) Rg(t) is the Riccati quotient in (2.7) associated with (X, U) on [a, oo) and the equality in (7.5) holds. Moreover, multiplying (7.5) by the matrix RAoa(a) from the both sides and using the identities RAoa(a) Rg(a) = -Raoo(oO = Rg(®) RAoo(a) yield Z = Z + RAao(a)X^T(a)NM-1X^(a)RAao(a). (7.7) In turn, by using (7.6) and (7.2) we have Z = Z and hence formula (7.7) becomes RAao(a) XtT(a) NM~1Xi(a) RAao(a) = 0. (7.8) From Remark 3.7 it follows that Inil'(a)JJAoo(a) = ImPj^, which means that we have X^(a) RAoo(a) K = Pg^ for some invertible matrix K. By using (7.8) we then obtain that PSooNM-1PSoo = KTRAoa(a) X]T(a) NM'1 X](a) RAoa(a) K (=8) 0, which is the last condition in (7.4). Thus, according to Proposition 3.14 the conjoined basis (X, U) is a principal solution of (H) at infinity with respect to [a, oo). Finally, with the aid of Remark 7.6 the matrix Q(i) is then a distinguished solution of (TV) at infinity with respect to [a, oo). The proof is complete. □ In the next three results we study the minimality of distinguished solutions of (TV). This minimality property needs to be understood in the following sense. For 1722 peter šepitka every symmetric solution Q(t) of (TV) there exists a distinguished solution of (TV), which exists on the same interval and is at the same time smaller than Q(t) on this interval (Theorems 7.16 and 7.18). On the other hand, any symmetric solution of (H), which is smaller than a distinguished solution of (H) on some interval, is a distinguished solution itself with respect to this interval (Theorem 7.20). However, in general there is no universal "smallest" distinguished solution of the Riccati equation (TV), see Remark 7.21. We also note that in the first result we consider the case when the solutions satisfy condition (1.4), while in the second and third result this assumption is removed. Theorem 7.16. Assume (1.1). Let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (3.11) and let Q(t) be a symmetric solution of the Riccati equation (TV) on [a, oo) C [«00, oo) such that inclusion (1.4) holds. Then there exists a distinguished solution Q(t) of (TV) at infinity with respect to [a, oo) satisfying (1.4) such that Q(t) > Q(t) for every t € [a, oo). Proof. Let Q(t) be as in the theorem and let (X, U) be its associated conjoined basis of (H) in Theorem 4.21 or Remark 4.23. Then (X, U) has constant kernel on [a, oo) and belongs to the genus Q. Moreover, through (1.4) the matrix Q(t) = Rg(t) Q(t) Rg(t) is the Riccati quotient in (2.7) corresponding to (X, U) on [a, oo). Let Ta be the T-matrix in (2.13) associated with (X,U) on [a, oo) and consider the solution (X, U) of (H) in (3.4). From Theorem 3.5 and Remark 3.6 we know that (X, U) is a principal solution of (H) at infinity belonging to the genus Q. Let Q(t) be its corresponding Riccati quotient in (2.7) on [a, oo). According to Theorem 7.5 and Remark 7.6 the matrix Q(t) is a distinguished solution of (7?.) at infinity with respect to [a, oo) satisfying condition (1.4). Moreover, since the matrix Ta is nonnegative definite, by Remark 2.5, with the aid of Corollary 5.5(h) with Q := Q, N := -Ta, and M := I we then have that Q(a) > Q(a). Finally, this inequality implies through Corollary 4.15 that Q(t) > Q(t) for all t € [a, oo), which completes the proof. □ Remark 7.17. (i) By applying (5.7) in Theorem 5.3 we obtain an exact relation between the Riccati quotients Q(t) and Q(t) on [a, oo). Namely, the formula holds for every t € [a, oo). In particular, for t = a the equality in (7.9) becomes (ii) According to Remark 7.14, the point a € [«00,00) in Theorem 7.16 satisfies a > am;n. Moreover, from Theorems 4.3 and 7.16 it follows that the last inequality holds even when condition (1.4) regarding the matrix Q(t) is dropped. Hence, we conclude that for any genus Q the open interval (am-ln, 00) is the maximal set such that there exists a symmetric solution of the Riccati equation (TV) on (am-ln, 00). Theorem 7.18. Assume (1.1). Let Q be a genus of conjoined bases of (H) with the orthogonal projector Rg(t) in (3.11). LetQ(t) be a symmetric solution of the Riccati equation (TV) on [a, 00) C [«00,00). Then there exists a distinguished solution Q(t) of (TV) with respect to [a, 00) such that Q(t) > Q(t) holds for every t € [a, 00). Proof. We proceed similarly as in the proof of Theorem 7.16. Let (X, U) be a conjoined basis of (H) from the genus Q, which corresponds to Q(t) on [a, 00) through Q(t) = Q(t) - X^T(ť)Ta [P - 5«(i)Ta]t X\t) (7.9) Q(a) = Q(a) - X]T(a)TaX](a). (7.10) riccati equations for linear hamiltonian systems 1723 Theorem 4.21. In particular, (X, U) has constant kernel on [a, oo) and the symmetric matrix Q*(t) := Rg(t) Q(t) Rg(t) is the associated Riccati quotient in (2.7) for every t € [a, oo). Moreover, according to formula (7.10) in Remark 7.17 with Q* := Q the solution Q*(t) of (TV) satisfying the condition 4(a) = Qt(a)-X^T(a)TaX^(a) = Rg(a) Q(a) Rg(a)-XtT(a) Ta X1"(a) (7.11) is a distinguished solution of (TV) at infinity with respect to [a, oo). Here Ta is the T-matrix in (2.13) associated with (X,U). Furthermore, let Da be the matrix in (4.19), which corresponds to Q(t) through the F-matrix Fa(t) in (4.18) on [a, oo), and consider the symmetric solution Q(t) of (7?.) given by initial condition Q(a) := Q(a) — Da. We will show that Q(t) is a distinguished solution of (7?.) at infinity with respect to [a, oo). Let RAoa(t) be the orthogonal projector defined in (3.7). Similarly, as in the proof of Theorem 7.15 we will use the notation Z := RAoo(a)Q(a) RAoo(a), Zt := RAoo(a) Qt(a) RAoo(a). (7.12) With the aid of (3.11) and Remark 5.2(h) together with the symmetry of the matrices RAoa(t), Rg(t), and Da we have the identities RAao(a) Rg(a) = RAoa(a) = Rg(a) RAoa(a) and RAoo(a) Da = Da = Da RAoo(a). By combining these properties with (7.11) and the second equality in (5.6) we obtain that Z ('7=2') RAoa(a) Q(a) RAoa(a) = RAoa(a) [Q(a) — Da] RAoa(a) = RAoa(a) Rg(a) Q(a) Rg(a) RAoa(a) - Da {7=X) RAoo (a) [ 4 (a) + X]T(a) Ta X] (a) ] RAoo (a) - Da {7=2) Zt + RAoa(a) XtT(a) Ta X\a) RAoa(a) - Da (=6) Zt. (7.13) Finally, since the point a > am-ln by Remark 7.17(h), from (7.12), the equation in (7.13), and Theorem 7.15 it follows immediately that the solution Q(t) is a distinguished solution of (7?.) at infinity with respect to [a, oo). In particular, the matrix Q(t) solves equation (TV) on the whole interval [a, oo). And since Q(a) — Q(a) = Da > 0, we conclude by Corollary 4.14 that the inequality Q(t) > Q(t) holds for every t € [a, oo), which completes the proof. □ Remark 7.19. We note that the converse to Theorem 7.18 also holds. More precisely, if Q(t) is a distinguished solution of (TV) at infinity with respect to the interval [a, oo), then every symmetric solution Q(t) of (TV), which satisfies the condition Q(a) > Q(a), exists on the whole interval [a, oo) and the inequality Q(t) > Q(t) holds on [a, oo). This observation is a direct application of Corollary 4.15 with the choice Q := Q and Q := Q. Theorem 7.20. Assume (1.1) and let Q be a genus of conjoined bases of (H) with the matrix Rg(t) in (3.11). Let Q(t) be a distinguished solution of the Riccati equation (TV) with respect to the interval [a, oo) C [«00,00). Moreover, let Q(t) be a symmetric solution of (TV) on [a, 00) satisfying the initial condition Q(a) > Q(a). Then Q(t) is a distinguished solution of (TV) at infinity with respect to [a, 00) and the inequality Q(t) > Q(t) holds for all t € [a, 00). Proof. Let Q(t) and Q(t) be as in the theorem. By using Corollary 4.14 we obtain the inequality Q(t) > Q(t) on [a, 00). On the other hand, according to Theorem 7.18 there exists a distinguished solution Q(t) of (TV) at infinity such that Q(t) > Q(t) 1724 peter šepitka for every t € [a, oo). Hence, for t = a we have the relations Q(a) > Q{a) > Q(a). Consequently, by multiplying the last inequalities by the matrix RAao (a) defined in (3.7) from the both sides we obtain that RAoo(a) Q(a) RAoo(a) > RAoo(a) Q(a) RAoo(a) > RAoo(a) Q(a) RAoo(a). (7.14) But Theorem 7.15 and the fact that both the solutions Q(t) and Q(t) are distinguished with respect to [a, oo) yield RAoa(a) Q(a) RAao(a) = RAoa(a) Q(a) RAao(a). Therefore, the inequalities in (7.14) are implemented as the equalities. In turn, applying Theorem 7.15 once more then implies that Q(t) is a distinguished solution of (TV) at infinity with respect to [a, oo) as well. The proof is complete. □ Remark 7.21. Given a genus Q of conjoined bases of (H) with the matrix Rg(t) defined in (3.11), let Q(t) be a distinguished solution of (TV) at infinity with respect to the interval [a, oo) C [«00,00). Then there exist distinguished solutions Q*(t) and Qtt(t) of (TV) satisfying Q*(t) am-ln. Hence, by applying Theorem 7.15 we obtain immediately that Q*(t) and Qtt(t) are distinguished solutions of (TV) at infinity with respect to [a, 00). In addition, since Q(a) — Qt(a) = I — RAao(a) = Q**(a) — Q(a) by (7.16) and / — RAao(a) > 0, we have the inequalities Qt(a) < Q(a) < Q**(a). In turn, according to Corollary 4.14 the inequalities in (7.15) hold. This observation shows that for the case of a general (not necessarily controllable) system (H) the partially ordered set of all distinguished solutions of (TV) has neither a minimal element nor a maximal element. The considerations in Theorems 7.15 and 7.16 show that for the minimal genus Qmim i-e-j f°r Rg(t) = RAx,(t), there exists a uniquely determined distinguished solution of the Riccati equation (TV) with A(t):=A(t)RAoo(t)-AT(t)[I-RAoo(t)], 1 B(t):=B(t), C(t):=RAoo(t)C(t)RAoo(t), J l' ' which represents the smallest element in the set of all symmetric solutions Q(t) of equation (TV) satisfying (1.4). Definition 7.22. Let Qm-ln be the minimal genus of conjoined bases of system (H) with the minimal orthogonal projector RAoa(t) in (3.7). A symmetric solution Q(t) of the Riccati equation (TV) with the coefficients in (7.17) is said to be a minimal distinguished solution at infinity if the matrix Q(t) is defined on some interval riccati equations for linear hamiltonian systems 1725 [a, oo) C [ttoo, oo) such that ImQ(t) C lmRAoo(t), fe[a,oo), (7.18) and its corresponding matrix Fa(t) in (4.18) satisfies F^(t) —> 0 as t —> oo. The following result shows the existence and uniqueness of the minimal distinguished solution of the Riccati equation (TV) for the minimal genus Qm-ln, as well as its minimality property. Theorem 7.23. Assume (1.1). Then system (H) is nonoscillatory if and only if there exists a minimal distinguished solution Q(t) of the Riccati equation (TV) with the coefficients in (7.17). In this case, the minimal distinguished solution Q(t) is determined uniquely and any symmetric solution Q(t) of (TV) on [a, oo) C [«00,00) with (7.18) satisfies Q(t) > Q(t) on [a, 00). Proof. The first part of the theorem coincides with the statement of Remark 7.10 for the genus Q = Qm-ln and for the corresponding orthogonal projector Rg(t) in (3.11) equal to the minimal orthogonal projector RAoa(t) defined in (3.7). The uniqueness of the distinguished solution Q(t) follows from Theorem 7.15 with Q := Qm-ln. More precisely, let Q(t) and Qt(t) be two distinguished solutions of equation (7?.) for the minimal genus Qm-ln, which satisfy condition (1.4) (with Rg(t) = RAoa(t)) on [a, 00) for some a > a^. From Theorem 7.13 it follows that both matrices Q(t) and Qt(t) solve (TV) on the maximal open interval (am;n,oo) and hence, the point a € (ciminjOo). According to Corollary 4.5 we then obtain the inclusions lvnQ(t) C lvnRAoa(t) and ImQt(t) C lvnRAoa(t) for every t € (amin,oo). Consequently, combining these facts with the symmetry of Q(t) and Qt(t) yields RAoo(t)Q(t)RAeo(t) = Q(t), RAoo(t)Q*(t)RAUt) = Q*(t) (7.19) for all t € (amin, oo)- Finally, by using formula (7.2) in Theorem 7.15 with Q := Qm-ln and Q := Qt together with (7.19) we obtain on (ctmin, 00) that Q(t){ =9) RAoo(t)Q(t)RAao(t) (7= RAUt)Q*(t)RAUt)( =9) Thus, the distinguished solutions Q(t) and Qt(t) coincide. Finally, the minimality property of the minimal distinguished solution of (TV) at infinity follows from Theorem 7.16. Namely, by using the latter reference for any symmetric solution Q(t) of (TV) on [a, 00) C [«00,00) with (7.18) there exists a distinguished solution Q(t) of (TV) at infinity with respect to [a, 00) satisfying (7.18) such that Q(t) > Q(t) for every t € [a, 00). In fact, the matrix Q(t) is the minimal distinguished solution of (TV) at infinity, by Definition 7.22. The proof is complete. □ Remark 7.24. The minimal distinguished solution of (TV) at infinity in Theorem 7.23 will be denoted by Qmin- The minimal distinguished solution Qmin plays for the theory of the Riccati differential equations (TV) or (R) a similar role as the minimal principal solution (Xm-ln, Um-ln) of system (H) at infinity for the theory of principal solutions at infinity. Remark 7.25. When system (H) is completely controllable, the main results of this section give the classical statements about the distinguished solutions at infinity of the Riccati equation (R). More precisely, the following holds. 1726 peter šepitka • The results in Corollary 7.7 and Theorem 7.15 yield the correspondence between the unique principal solution of (H) at infinity and the unique distinguished solution of (R) at infinity, see [7, pg. 53] or [23, pp. 45-46]. • The result in Theorem 7.8 provides a characterization of the nonoscillation of system (H) in terms of the existence of the unique distinguished solution of (R) at infinity, see the necessary condition in [22, Theorem VII.3.3]. Note that the nonoscillation of (H) is defined in [22, Section VII.3] in terms of disconjugacy of (H), i.e., in terms of the nonexistence of mutually conjugate points, which is a stronger concept than the nonoscillation of (H) as we define in Section 2. We note also that the sufficiency part of Theorem 7.8 is new also in the completely controllable case. • The results in Theorems 7.18 and 7.20 yield the minimality property of the unique distinguished solution of (R) at infinity, see [7, Theorem 8, pg. 54] or [23, Theorem IV.4.2]. Indeed, in this case rfoo = 0 and there is only one minimal/maximal genus of conjoined bases of (H). This implies that = a and the orthogonal projector RAoa(t) in (3.7) satisfies RAoa(t) = I on [a, oo). Therefore, the unique Riccati equation (7?.) associated with the minimal/maximal genus coincides with the classical Riccati equation (R). Moreover, under the Legendre condition (1.1) the nonoscillation of system (H) is then equivalent with the existence of a unique (minimal) distinguished solution Q of (R) at infinity. In addition, the matrix Q constitutes the smallest symmetric solution of the Riccati equation (R), that is, every symmetric solution Q of (R) on [a, oo) C [a, oo) satisfies inequality (1.5). Remark 7.26. We note that the results commented on in Remark 7.25 hold under a weaker assumption than the complete controllability of (H). More precisely, under (1.1) and the nonoscillation of (H) the existence of a unique distinguished solution at infinity of a (unique) Riccati equation (1Z) is equivalent with the fact that the maximal order of abnormality rfoo = 0. 8. Examples. In this section we provide several examples which illustrate the presented theory of Riccati equations for abnormal system (H). Example 8.1. In the first example we explore a controllable linear Hamiltonian system. For n = 1, a = 0 we consider system (H) with A(t) = 0, B(t) = 1 +t2, and C(t) = —2/(1 + i2)2. This system comes from the second order Sturm-Liouville equation [y'/(l + t2)]' + 2y/(l + t2)2 = 0. The matrix B(t) > 0 on [0,oo), which implies that system (H) is completely controllable on [0, oo) with d[0, oo) = rfoo = 0 and = 0, by (2.19). Thus, there exists only one (minimal/maximal) genus Q of conjoined bases with the corresponding orthogonal projector Rg(t) e 1 on [0, oo) and consequently, the unique Riccati equation Q' + (l + t2)Q2+2/(l + t2)2 = 0, te[0,oo). (8.1) In [30, Example 7.1] we showed that system (H) is nonoscillatory and that the principal solutions at infinity are nonzero multiples of (X(t),U(t)) = (t, 1/(1+ i2)), with dmin = 0, by (3.1). Therefore, by Theorem 7.5 and Remark 7.25 the unique (minimal) distinguished solution Q of (8.1) at infinity satisfies Q(t) = l/[t(l + t2)]; te(0,oo). riccati equations for linear hamiltonian systems 1727 Moreover, by using Proposition 4.12 with Q := Q the general solution Q(-,a,p) of the Riccati equation (8.1) defined on an interval [a, oo) C (0, oo) has the form Q(t,a,p) = Q(t) +p/[t2 +pt(t - a)(t+ l/a)], fe[a,oo), (8.2) with p € [0, oo) U {oo}, where Q(t, a, oo) := lim Q(t, a,p) (=2) Q(t) + l/[t(t - a)(t + l/a)], t e [a, oo). (8.3) p—>oo From formulas (8.2)-(8.3) it then follows that for any point a > 0 and parameter p € [0, oo) U {oo} we have the inequality Q(t, a,p) > Q(t) on [a, oo), as we claim in Theorem 7.18 or Remark 7.25. Example 8.2. In this example we consider the so-called zero system (H) with n x n coefficient matrices A(t) = B(t) = C(t) = 0 on [a, oo). This system is nonoscillatory and extremely abnormal, that is, d[a, oo) = rfoo = n and hence, "oo = o, and RAao(t) = 0 on [a, oo). In [29, Example 8.2] and [31, Example 5.7] we showed that every conjoined basis of (H) is a constant principal solution at infinity with respect to [a, oo) and that the set of all genera of conjoined bases of (H) is isomorphic to the complete lattice of all subspaces in R™. Namely, for every constant orthogonal projector R € R™x™ there exists a unique genus Q of conjoined bases of (H) such that Rg(t) = R on [a, oo). The associated Riccati equation (TV) reduces to Q' = 0. In this case, every symmetric solution of (TV) is a constant distinguished solution at infinity with respect to the interval [a, oo), so that dmin = a. In particular, for any constant symmetric matrix M € R™x™ the pair (X, U) := (R, MR + I — R) constitutes a constant principal solution of (H) at infinity belonging to Q, which corresponds to the distinguished solution Q(t) = M of (TV) on [a, oo) via Theorems 4.21 and 7.4. Moreover, the minimal distinguished solution at infinity satisfies Qm-ln(t) = 0 on [a, oo). In the previous two examples we studied the situation when system (H) possessed only one Riccati equation (TV). However, this will be not the case of the system presented in the last example. Example 8.3. For n = 3 and a = 0 we consider system (H) with the coefficients A(t) = diag{0, 0,1}, B(t) = diag{l + t2, 0, 0}, and C(t) = diag{-2/(l + t2)2, 0,0} on [0,oo). In this case we have rfoo = 2, = 0, and RAoa(t) = diag{l,0,0} on [0, oo). Moreover, in [31, Example 5.8] we examined the set of all genera Q of conjoined bases of (H) and found a principal solution at infinity in each genus Q. We will continue in this study by illustrating the concept of distinguished solutions at infinity of the associated Riccati equations (TV). For the minimal genus Q = Qm-ln represented by the orthogonal projector Rg(t) = RAoo(t) on [0, oo) we have the Riccati equation (TV) with the coefficients in (7.17), i.e., Amin(t) = -A(t), Bmin(t) = B(t), Cmin(t) = C(t), t e [0, oo). This Riccati equation possesses on (0, oo) the minimal distinguished solution Qmin(t) = diag{l/[t(l + t2)], 0, 0} (8.4) constructed from the minimal principal solution (Xmin(t),L7min(t)) = (diag{t,0,0}, diag{l/(l + t2),l,e-*}) of (H) at infinity via Theorem 4.18, as well as the distinguished solution Q0(t) = diag{l/[t(l + i2)], 1, -e2*}, (8.5) 1728 peter šepitka which does not satisfy condition (7.18). In particular, the distinguished solutions in (8.4) and (8.5) are mutually incomparable on the interval (0, oo). Similarly, with the maximal genus Q = £/max represented by the orthogonal projector Rg(t) = I on [0, oo) there is associated the Riccati equation (R) with the pair of incomparable distinguished solutions at infinity Q(t) = diagjf/[t(l + t2)},-1, e"2*}, Q.(t) = diagjf/[t(l + t2)}, 1, -e"2*} (8.6) for t € (0, oo). We note that Q(t) and Q*(t) in (8.6) are both the Riccati quotients in (2.7), which correspond to the maximal principal solutions (X(t),U(t)) = (diag{t,f,e*}, diagjf/(l + i2),-I, e"*} ), (X,(t),U,(t)) = (diag{t,f,e*}, diagjf/(l + i2), I, -e"*} ) of (H) at infinity, respectively. In the remaining part of this example we analyze for three different genera with rank equal to r = 2 the corresponding Riccati equations (TV) and their distinguished solutions. More precisely, according to [31, Example 5.8] we consider the genera Qi, Q2, and Q3 given by R0l (t) = diag{f, 0, f}, R52(t) = diag{f, f, 0}, /e2* + f 0 0 \ ™) = *n( : I- $) on [0, 00). With the genus Q\ we associate the Riccati equation (TV) with Ai(t)=A(t), Bl(t) = B(t), Ci(t) = C(t), te[0,oo), possessing the pair of incomparable distinguished solutions at infinity Qi(t) = diagjf/[t(l + t2)}, 0, -e"2*}, Qlt(t) = diagjf/[t(l + t2)},-1, 0} for (0,oo). The matrix Qi(t) is the Riccati quotient in (2.7), which corresponds to the principal solution (Xi(t),C>i(t)) = (diag{i,0,e*}, diag{f/(f+ t2),f,-e-*}) of (H) at infinity belonging to Qi, while the distinguished solution Qi*(t) does not satisfy (f .4). Similarly, for the genus Q2 we have the Riccati equation (TV) with A2(t) = -A(t), B2(t) = B(t), C2(t) = C(t), t e [0,00), which has the pair of incomparable distinguished solutions at infinity Q2(t) =diag{f/[t(f + i2)],f,0}, Q2*(t) = diag{f/[t(f + i2)],0,e2t} for (0,oo). The matrix Q2(t) is the Riccati quotient in (2.7), which corresponds to the principal solution (X2(t),U2(t)) = (diag{i,f,0}, diag{f/(f+ t2),f,e-*}) of (H) at infinity from the genus Q2 and the distinguished solution Q2*(t) does not satisfy (f .4). Finally, for the genus Q3 we obtain the Riccati equation (7?.) with Aa(t) = ——- 0 0 0 , Ba(t) = B(t), Ca(t) = C(t), t e [0,oo), riccati equations for linear hamiltonian systems 1729 having on (0, oo) the pair of incomparable distinguished solutions at infinity In particular, the matrix Q%(t) constitutes the Riccati quotient in (2.7) associated with the principal solution at infinity Acknowledgments. The author is grateful to Professor Roman Simon Hilscher for consultations regarding the subject of this paper. [1] J. Allwright and R. Vinter, Second order conditions for periodic optimal control problems, Control Cybernet, 34 (2005), 617-643. [2] F. V. Atkinson, Discrete and Continuous Boundary Problems, Academic Press, New York - London, 1964. [3] R. Bellman, Introduction to the Mathematical Theory of Control Processes. Vol. I: Linear Equations and Quadratic Criteria, Mathematics in Science and Engineering, Vol. 40, Academic Press, New York, NY, 1967. [4] A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory and Applications, Second Edition, Springer-Verlag, New York, NY, 2003. [5] D. S. Bernstein, Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory, Princeton University Press, Princeton, 2005. [6] S. L. Campbell and C. D. Meyer, Generalized Inverses of Linear Transformations, Reprint of the 1991 corrected reprint of the 1979 original, Classics in Applied Mathematics, Vol. 56, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2009. [7] W. A. Coppel, Disconjugacy, Lecture Notes in Mathematics, Vol. 220, Springer-Verlag, Berlin - New York, 1971. [8] Z. Došlá and O. Došlý, Quadratic functionals with general boundary conditions, Appl. Math. Optim., 36 (1997), 243-262. [9] P. Hartman, Ordinary Differential Equations, John Wiley, New York, 1964. [10] M. R. Hestenes, Quadratic control problems, J. Optim. Theory Appl., 17 (1975), 1—42. [11] R. Hilscher and V. Zeidan, Applications of time scale symplectic systems without normality, J. Math. Anal. Appl, 340 (2008), 451-465. [12] R. Hilscher and V. Zeidan, Time scale embedding theorem and coercivity of quadratic functionals, Analysis (Munich), 28 (2008), 1-28. [13] R. Hilscher and V. Zeidan, Riccati equations for abnormal time scale quadratic functionals, J. Differential Equations, 244 (2008), 1410-1447. [14] R. Johnson, C. Nunez and R. Obaya, Dynamical methods for linear Hamiltonian systems with applications to control processes, J. Dynam. Differential Equations, 25 (2013), 679—713. [15] R. Johnson, R. Obaya, S. Novo, C. Nunez and R. Fabbri, Nonautonomous Linear Hamiltonian Systems: Oscillation, Spectral Theory and Control, Developments in Mathematics, Vol. 36, Springer, Cham, 2016. [16] R. E. Kalman and R. S. Bucy, New results in linear filtering and prediction theory, Trans. ASME Ser. D J. Basic Engrg., 83 (1961), 95-108. and the distinguished solution Q%*{t) does not satisfy condition (1.4). REFERENCES 1730 peter šepitka [17] W. Kratz, Quadratic Functionals in Variational Analysis and Control Theory, Akademie Verlag, Berlin, 1995. [18] W. Kratz, Definiteness of quadratic functionals, Analysis, 23 (2003), 163—183. [19] B. P. Molinari, Nonnegativity of a quadratic functional, SIAM J. Control, 13 (1975), 792-806. [20] W. T. Reid, Riccati matrix differential equations and non-oscillation criteria for associated linear differential systems, Pacific J. Math., 13 (1963), 665—685. [21] W. T. Reid, Principal solutions of nonoscillatory linear differential systems, J. Math. Anal. Appl, 9 (1964), 397-423. [22] W. T. Reid, Ordinary Differential Equations, John Wiley & Sons, Inc., New York — London - Sydney, 1971. [23] W. T. Reid, Riccati Differential Equations, Academic Press, New York — London, 1972. [24] W. T. Reid, Sturmian Theory for Ordinary Differential Equations, Springer-Verlag, New York - Berlin - Heidelberg, 1980. [25] V. Růžičková, Discrete Symplectic Systems and Definiteness of Quadratic Functionals, Ph.D thesis, Masaryk University, Brno, 2006. Available from https://is.muiii.cz/th/p9iz7/ ?lang=en. [26] P. Sepitka, Theory of Principal Solutions at Infinity for Linear Hamiltonian Systems, Ph.D thesis, Masaryk University, Brno, 2014. Available from https://is.muiii.cz/th/vqad7/ ?lang=en. [27] P. Sepitka, Genera of conjoined bases for (non)oscillatory linear Hamiltonian systems: extended theory, submitted, 2017. [28] P. Sepitka and R. Simon Hilscher, Minimal principal solution at infinity for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations, 26 (2014), 57—91. [29] P. Sepitka and R. Simon Hilscher, Principal solutions at infinity of given ranks for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations, 27 (2015), 137—175. [30] P. Sepitka and R. Simon Hilscher, Principal and antiprincipal solutions at infinity of linear Hamiltonian systems, J. Differential Equations, 259 (2015), 4651—4682. [31] P. Sepitka and R. Simon Hilscher, Genera of conjoined bases of linear Hamiltonian systems and limit characterization of principal solutions at infinity, J. Differential Equations, 260 (2016), 6581-6603. [32] P. Sepitka and R. Simon Hilscher, Reid's construction of minimal principal solution at infinity for linear Hamiltonian systems, in Differential and Difference Equations with Applications (eds. S. Pinelas, Z. Došlá, O. Došlý and P. E. Kloeden), Proceedings of the International Conference on Differential & Difference Equations and Applications (Amadora, 2015), Springer Proceedings in Mathematics & Statistics, Vol. 164, Springer, (2016), 359—369. [33] R. Simon Hilscher, On general Sturmian theory for abnormal linear Hamiltonian systems, in Dynamical Systems, Differential Equations and Applications (eds. W. Feng, Z. Feng, M. Grasselli, A. Ibragimov, X. Lu, S. Siegmund and J. Voigt), Proceedings of the 8th AIMS Conference on Dynamical Systems, Differential Equations and Applications (Dresden, 2010), Discrete Contin. Dynam. Systems, Suppl. 2011, American Institute of Mathematical Sciences (AIMS), (2011), 684-691. [34] J. L. Speyer and D. H. Jacobson, Primer on Optimal Control Theory, Advances in Design and Control, Vol. 20, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2010. [35] G. Stefani and P. Zezza, Constrained regular LQ-control problems, SIAM J. Control Optim., 35 (1997), 876-900. [36] M. Wahrheit, Eigenvalue problems and oscillation of linear Hamiltonian systems, Int. J. Difference Equ., 2 (2007), 221-244. [37] V. Zeidan, Sufficiency criteria via focal points and via coupled points, SIAM J. Control Optim., 30 (1992), 82-98. [38] V. Zeidan, The Riccati equation for optimal control problems with mixed state-control constraints: necessity and sufficiency, SIAM J. Control Optim., 32 (1994), 1297-1321. [39] V. Zeidan, New second-order optimality conditions for variational problems with C2-Hamiltonians, SIAM J. Control Optim., 40 (2001), 577-609. Received August 2017; revised July 2018. E-mail address: sepitkap@math.muiii.cz APPENDIX C Paper by Sepitka & Simon Hilscher (JDE 2017) This paper entitled "Comparative index and Sturmian theory for linear Hamiltonian systems" appeared in the Journal of Differential Equations, 262 (2017), no. 2, 914-944, see item [84] in the bibliography. 109 Available online at www.sciencedirect.com (D ScienceDirect Journal of CrossMark Differential Equations J. Differential Equations 262 (2017) 914-944 www.elsevier.com/locate/jde Comparative index and Sturmian theory for linear Hamiltonian systems * v v Peter Sepitka, Roman Simon Hilscher * Department of Mathematics and Statistics, Faculty of Science, Masaryk University, Kotlářská 2, CZ-61137Brno, Czech Republic Received 27 July 2016; revised 27 September 2016 Available online 6 October 2016 Abstract The comparative index was introduced by J. Elyseeva (2007) as an efficient tool in matrix analysis, which has fundamental applications in the discrete oscillation theory. In this paper we implement the comparative index into the theory of continuous time linear Hamiltonian systems, study its properties, and apply it to obtain new Sturmian separation theorems as well as new and optimal estimates for left and right proper focal points of conjoined bases of these systems on bounded intervals. We derive our results for general possibly abnormal (or uncontrollable) linear Hamiltonian systems. The results turn out to be new even in the case of completely controllable systems. We also provide several examples, which illustrate our new theory. © 2016 Elsevier Inc. All rights reserved. MSC: 34C10 Keywords: Linear Hamiltonian system; Sturmian separation theorem; Proper focal point; Comparative index; Conjoined basis; Controllability 1. Introduction In this paper we study oscillation properties of solutions of the linear Hamiltonian system * This research was supported by the Czech Science Foundation under grant GA16-0061 IS. Corresponding author. E-mail addresses: sepitkap@math.muni.cz (P. Sepitka), hilscher@math.muni.cz (R. Simon Hilscher). http://dx.doi.Org/10.1016/j.jde.2016.09.043 0022-0396/© 2016 Elsevier Inc. All rights reserved. P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 915 x' = A(t)x + B(t)u, u' = C(t)x - A1\t)u, te[a,b], (H) where A, B,C : [a, b] —>- M.nxn are given piecewise continuous matrix-valued functions on the interval [a, b] such that B(t) and C(t) are symmetric and the Legendre condition holds, i.e., B(t) >0 for all? e [a,b]. (1.1) Here n e N is a given dimension and a, b e M, a < b, are fixed numbers. The main results of this paper are concerned with the Sturmian type separation theorems about the number of focal points of conjoined bases of (H) in the given interval. We present a novel approach to this problem, which is based on the so-called comparative index of two conjoined bases of (H), see (2.3) in Section 2 below. System (H) is traditionally studied under the complete controllability assumption. This means that the only solution (x, u) of (H) with x(t) = 0 on a subinterval of [a, b] with positive length is the trivial solution (x, u) = 0 on [a, b], see e.g. [5,17,22,26,27]. In this case to e [a, b] is afocal point of a conjoined basis (X, U) of (H) if X(to) is singular, and then m(to) := def X(to) = dimKerX(fo) is its multiplicity. We refer to Section 3 for the definition of a conjoined basis. Every conjoined basis of (H) then has finitely many focal points in [a,b], and the numbers of focal points in (a, b] or in [a, b) of any two conjoined bases of (H) differ by at most n, see [22, Theorem 4.1.3, p. 126] and [26, Corollary 1, p. 366]. In addition, for one conjoined basis (X, U) of (H) the difference between the numbers of its focal points in (a, b] and in [a, b) equals the value def X(b) - def X (a) = rank X(a) - rank X(b). (1.2) When the controllability assumption is absent, Kratz showed in [23, Theorem 3] the following crucial result. Proposition 1.1. Assume that (1.1) holds. Then for any conjoined basis (X, U) of(H) the kernel of X(t) is piecewise constant on [a, b], i.e., there is a partition a = to < t\ < ■ ■ ■ < tm = b such that KerX(t) is constant on the open interval (tj, tj+\) for all j e {0, 1,..., m — 1} and KerX(t7) QKer X(tj), j e {1,2,..., m), (1.3) J J KerX(f+) cKerX(?/), j e {0,1,..., m - 1}. (1.4) J J The quantity KerX(fjt) denotes the limit of the constant set KerX(t) as t —>- tj1. The inclusions in (1.3) and (1.4) follow from the continuity of X(t) on [a,b]. In the subsequent work [36], Wahrheit defined the point to e (a,b] to be a left proper focal point of (X, U) if KerX(f0") ^ KerX(fo), with the multiplicity mL(to) :=def X(to) - def X(t^) =rankX(fQ") -rankX(fo). (1.5) In a similar way we define to e [a, b) to be a right proper focal point of (X, U) by the condition KerX(fQh) ^ KerX(fo), with the multiplicity 916 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 mR(t0) :=def X(t0) -defA^) = rank X(?0+) -rank X(t0). (1.6) The notations def X(t^) and rankX(f^) represent the one-sided limits at to of the piecewise constant quantities def X(t) and rankX(f). The above mentioned Sturmian separation theorem by Reid in [26, Corollary 1, p. 366] was generalized to possibly abnormal (or uncontrollable) system (H) in [24, Corollary 4.8] and [34, Theorems 1.4 and 1.5] by Kratz and the second author. We review these statements below for our future reference in this paper. Recall that a principal solution (Xs, Us) at the point s e [a, b] is defined as the solution of (H) with the initial conditions Xs(s) = 0 and Us(s) = I, see also (3.1). We stress that the focal points are always counted including their multiplicities. Proposition 1.2. Assume (1.1) and let m e N U {0} be fixed. The principal solution (Xa, Ua) of (H) has m left proper focal points in (a, b] if and only if the principal solution (Xb, Ub) of(H) has m right proper focal points in [a, b). Proposition 1.3. Assume that (1.1) holds. If the principal solution (Xa, Ua) of (H) has m left proper focal points in (a, b], then any other conjoined basis (X, U) of(H) has at least m and at most m An left proper focal points in (a, b]. Similarly, if the principal solution (Xb, Ub) of(H) has m right proper focal points in [a, b), then any other conjoined basis (X, U) of (H) has at least m and at most m An right proper focal points in [a, b). Proposition 1.4. Assume that (1.1) holds. Then the difference between the numbers of left proper focal points in (a,b] of any two conjoined bases of (H) is at most n. Similarly, the difference between the numbers of right proper focal points in [a, b) of any two conjoined bases of (H) is at most n. We note that the statements in Propositions 1.3 and 1.4 regarding the right proper focal points follow from the corresponding results for the left proper focal points by the time-reversing transformation t i-> a + b — t, which is described in [24, Remark 4.7]. Linear Hamiltonian systems (H) without the complete controllability assumption are intensively studied in the literature. As we mentioned above, the second author derived in [34,35] general Sturmian separation and comparison theorems for such systems (H). Recently, Johnson, Novo, Nunez, and Obaya proved in [20, Theorem 3.6] a very nice formula connecting the rotation number of system (H) with the number of left proper focal points of (X, U) in (a, b] when b —>- oo. Uncontrollable systems (H) were also considered in [18,19,21] in the relation with the notion of a weak disconjugacy of (H) and dissipative control processes, and in [25,28-33] when studying the principal solutions of (H) at infinity. In the present paper we derive (Theorem 4.1) new explicit formulas for the difference of the left proper focal points of two conjoined bases of (H) in (a,b], and the right proper focal points in [a, b), as well as optimal bounds for the numbers of left and right proper focal points of one conjoined basis in a bounded interval (Theorems 5.2, 5.3, 5.6, and 5.9 and Corollaries 5.8 and 5.10). These estimates essentially improve the statements in Propositions 1.3 and 1.4. In addition, we establish (Theorem 5.1) an exact relationship between the numbers of left proper focal points in (a, b] and right proper focal points in [a, b) of one conjoined basis of (H), which generalizes the formula in (1.2) to abnormal systems. We note that these results are new even in the case of a completely controllable system (H). P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 917 As a main tool in the above study we use the comparative index, which was introduced by El-yseeva in [12,13] and successfully applied in the discrete oscillation theory, see [6-10,14,16]. In this paper we study the properties of the comparative index from a point of view of the continuous time linear Hamiltonian system (H). In particular, we obtain the continuity and limit properties of the comparative index (Theorems 6.1 and 6.5) and their relationship with the multiplicities of left and right proper focal points at a given point (Theorem 6.3). In a sense, some results in this paper can be regarded as continuous time analogs of the known results for symplectic difference systems in [13], see Remark 7.4. This paper therefore provides a key step in the implementation of recent discrete time methods into the new continuous time theory. The paper is organized as follows. In Section 2 we define the comparative index and display its algebraic properties, which are needed in this paper. In Sections 3 we recall some general facts about linear Hamiltonian systems and prove two auxiliary results about conjoined bases of (H). In Section 4 we derive key equalities relating the difference of the numbers proper focal points of two conjoined bases of (H) and the comparative index. In Sections 5 and 6 we establish further Sturmian separation theorems for conjoined bases of system (H) and the continuity and limit properties of the comparative index. Finally, in Section 7 we provide several examples illustrating our new theory, as well as comments about the related topics and future research directions. 2. Algebraic properties of comparative index In this section we present the definition of the comparative index of two matrices and its main properties from [12,13]. Let Y and Y be real constant 2n x n matrices such that YTJY = 0, YTJY = 0, rankF = n =rankf, W:=YTJY, J := (2.1) The matrix J is the canonical skew-symmetric matrix of dimension 2n. The first three conditions in (2.1) can be regarded as suitable initial conditions for conjoined bases of system (H), while the matrix W is the Wronskian of Y and Y. When we split Y and Y into n x n blocks Y = (XT, UT)T and Y = (XT, UT)T, then the first, second, and fourth expressions in (2.1) have the form XTU = UTX, XTU = UTX, W = XTU -UTX. (2.2) When considering the matrices Y and Y satisfying (2.1), we will always express them in the above block structure with X, U, X, U as in (2.2). Following [12, Definition 2.1] or [13, Definition 2.1], we define the comparative index /x(F, Y) and the dual comparative index /x*(F, Y) of Y and Y as the numbers lx(Y, Y) := rank+ indV, /x*(F, Y) :=rankA4 +ind(—V), (2.3) where M. and V are the n x n matrices M:=(I - XfX)W, V:= VWTXfXV, V := I - MfM, (2.4) 918 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 and where W is the Wronskian of Y and Y defined in (2.1). The dagger in (2.4) denotes the Moore-Penrose pseudoinverse of a matrix, i.e., the unique matrix X^ satisfying the four properties XfX and XXf are symmetric, XXfX = X, and XfXXf = XK We refer to [1,2,4] for general theory of pseudoinverse matrices. We note that the matrix V in (2.4) is the orthogonal projector onto Ker.M and that the matrix V is symmetric, see [13, Theorem 2.1]. In this case indP denotes the number of negative eigenvalues of V, and obviously we have ind(-V) = rankP - indV. For convenience we define the elementary 2n x n matrix E:=(0, if, (2.5) where 0 is the n x n zero matrix and I is the n x n identity matrix. The following properties in Propositions 2.1 and 2.2 are proven in [13, Section 2]. Proposition 2.1. Let Y and Y be 2n x n matrices satisfying (2.1) and let E be given by (2.5). Then the comparative index /x(F, Y) and the dual comparative index [i*(Y, Y) defined in (2.3) have the following properties: /x(Y, F) + rankX =/x*(F,F)+rank 1, (2.6) fi(Y, Y) + /x(F, Y) = rank W = /x*(F, Y) + /x*(F, Y), (2.7) /x(Y, Y) < minfrank W, rankX}, fi*(Y, Y) < min{rank W, rankX}, (2.8) fi(Y,E)=0 = [i*(Y,E), [i(E, Y) = rankX = /jl*(E, Y). (2.9) We note that the second conditions in (2.7) and (2.8) about the dual comparative index are not explicitly stated in [13], but they follow from (2.6) and from the corresponding properties for fi(Y, Y) in (2.7) and (2.8). We recall that a real 2n x 2n matrix S is symplectic if STJS = J. Symplectic matrices are of basic importance for the theory of linear Hamiltonian system (H), since any fundamental matrix of (H) is symplectic on [a, b] whenever it is symplectic at some initial point. Proposition 2.2. Let Z, Z, O be real 2n x 2n symplectic matrices and let E be defined by (2.5). Then the following transformation formulas hold: H(&ZE, ZE) - \x(ZE, ZE) = n(&ZE, &E) - \i(ZE, 0 and q > 0. We know from (2.6) and /x*(F, Y) > 0 that /x(F, Y) > rankl - rankA holds, and similarly from (2.6) and /x(F, Y) > 0 we get /x*(F, F) > rankA — rankA. We will analyze the latter difference. If rankA > rankA, then p = rankA — rankA, while if rankA < rankA, then p = 0. But since /x(F, Y) > 0 and /x*(F, Y) > 0, it follows from the above that [i(Y, Y) > p and /x*(F, Y) > p, i.e., (2.12) holds. Next, the first conditions in (2.7) and (2.8) imply that fi(Y, Y) (2= rank W - fi(Y, Y) {2> rank W - min{rank W, mnkX} = q. In a similar way we obtain from the second conditions in (2.7) and (2.8) that /x*(F, Y) > q. Therefore, the estimates in (2.13) hold as well. □ Applications of the comparative index in the oscillation theory of discrete symplectic systems can be found in [6-10,14,16]. In Section 4 below we will show how the comparative index arises in the theory of continuous time linear Hamiltonian system (H) and how it can be utilized in order to derive new Sturmian separation theorems for conjoined bases of (H). 3. Properties of linear Hamiltonian systems A 2n x n solution (A, U) of (H) is called a conjoined basis if XT (to) U(to) is symmetric and rank(Ar(?o), UT(to)) = n for some, and hence for any, point to e [a, b], compare with (2.1). As an example of a conjoined basis of (H) we mention the principal solution at the point s e [a, b], denoted by (Xs, Us), which is defined as the solution of (H) with the initial conditions Xs(s) = 0, Us(s) = I. (3.1) By [22, Corollary 3.3.9], any given conjoined basis (A, U) can be completed by another conjoined basis (A, U) to a (symplectic) fundamental matrix of (H), i.e., to a so-called normalized pair of conjoined bases, which then satisfy the relation W := XT (t) U(t) — UT (t) X(t) = I on [a,b], compare with (2.2). The fact that the fundamental matrix O := of (H) is symplectic is also equivalent with XUT - XUT = I, XXT=XXT, UUT = UUT (3.2) on [a,b], see [22, Proposition 1.1.5]. Moreover, similarly to [29-32] we say that (A, U) has constant kernel on the open or half-open interval (a,b), (a,b], [a,b), if the kernel of X(t) is constant on that interval. Let us fix for a moment a conjoined basis (A, U) of (H) with constant kernel on (a, b) and a point a e (a, b). We define the constant orthogonal projector P onto ImAr(f) = [Ker A(t)]±, the symmetric n x n matrix function S(t), and the orthogonal projector Ps(t) onto ImS(t) by P = P(t):=Xf(t)X(t), te(a,b), (3.3) t S(t):= J Xf(s)B(s)XfT(s)ds, te(a,b), (3.4) a Ps(t):=s\t)S(t) = S(t)S\t), te(a,b). (3.5) 920 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 We note that S(t) is correctly defined on (a,b), since in the present setting the function X^(t) is piecewise continuously differentiable on (a,b), hence continuous, see [4, Theorems 10.5.1 and 10.5.3]. The next result follows from standard properties of symmetric monotone matrix-valued functions. Lemma 3.1. Assume that (1.1) holds, (X, U) is a conjoined basis of (H) with constant kernel on (a, b), and a e (a, b) is given. Then the following hold. (i) The matrix S(t) < 0 on (a, a] and S(t) > 0 on [a, b). (ii) The matrix S(t) is nondecreasing and piecewise continuously differentiable on (a, b) with S'(t)=xHt)B(t)X^T(t)>0on (a,b). (iii) The set Im S(t) is nonincreasing on (a, a] and nondecreasing on [a,b) with Im S(t) = Im Pg(t) clmPon(a, b), and the following limits exist PSa ■= lim Ps(t), Im?5flclmP, (3.6) PSb := \im Ps(t), ImP^clrnP. (3.7) (iv) The matrix S^(t) is nonincreasing on (a, a + e) and on (b — e, b) for some e e (0, b — a) and the following limits exist Ta := lim 5f (0, Ta < 0, Im Ta c Im PSa, (3.8) Tb := lim Sf(t), Tb > 0, ImTb c ImPSfc. (3.9) t^b~ Proof. The statements in parts (i) and (ii) follow by direct considerations from the definition of S(t) in (3.4). The monotonicity property of ImS(t) in part (iii) is proven in [29, Theorem 4.2], which then yields that the set ImS(t) and hence the matrix Ps(t) are constant in some right neighborhood of a and in some left neighborhood of b. This shows that the limits in (3.6)-(3.7) exist. Part (iv) follows from the fact that the image of S(t) is constant on some right neighborhood (a, a + e) of a and on some left neighborhood (b — e, b) of b by part (iii), and S^(t) is nonincreasing on these neighborhoods by part (ii). Therefore, the limits in (3.8)-(3.9) indeed exist and have the stated properties. □ We note that the matrices P$a and P$i, in (3.6)-(3.7) are the maximal orthogonal projectors onto ImS(t) on (a, a] and on [a,b), respectively. Next we relate the matrix S(t) with some special conjoined basis (X, U) of (H) associated with (X, U) and a e (a, b). Lemma 3.2. Assume that (X, U) is a conjoined basis of (H) with constant kernel on (a, b) and a G (a, b) is given. Let the matrices P, S(t), Ps(t) and Ta, Tb be defined by (3.3)—(3.5) and (3.8)-(3.9). Then there exists a conjoined basis (X, U) of(H) such that (X, U) and (X, U) are normalized and x\a)X{a) = Q. (3.10) Moreover, the matrix X(t) also satisfies for all t e (a, b) the identities P. Šepitka, R. Šimon Hilscher / J. Differential Equations 262 (2017) 914-944 921 S(t) = Xf(t)X(t)P, X(t) P = X(t)S(t), (3.11) P(t) := X\t) X{t) = I-P + Ps(t), (3.12) Sf(i) = Af(i) X(t) P(t) = Af(i) X(t) Ps(t), (3.13) X(t)Sf(t)XT(t) = X(t)XT(t). (3.14) If in addition condition (1.1) holds, then X(a)XT(a) = X(a)TaXT(a), X (b) XT (b) = X(b) Tb XT (b), (3.15) X(t)XT(t) <0on [a,a], X(t) XT (t) > 0 on [a, b]. (3.16) Proof. The existence of a conjoined basis (X, U), which is normalized with (X, U) and which satisfies (3.10) and (3.11) is proven in [29, Theorem 4.4 and Remark 4.5(ii)]. We note that condition (1.1) is not needed in the assumption of the latter reference. By [28, Theorem 2.2.1 l(i)] we know that Ker A(?) = Im[P — Ps(t)] on (a, b), which means that the matrix I — P + Ps(t) is the orthogonal projector onto the set ImXT(t), i.e., identity (3.12) holds. Next, by the identities (suppressing the argument t) X = XP, PP$ = P$ = PsP, and PsS** = S** we obtain xfxP (3=2) xfxp (i -p + ps) = xfxps (3=5) xfxssf (3=1} XfXPSf (3A2) ppsf = y _ p + p^ psf = pspSi = psSf = sf ? which proves the identities in (3.13). In turn, we have by the symmetry of XXT, see (3.2), xsfxT (3=3) XXfXPXT = XXfXXT = XXfXXT = XXT = XXT, which shows that identity (3.14) also holds. Upon taking the limit as t a+ and t b~ in (3.14) and using the definition of Ta, Tb in (3.8)-(3.9) and the continuity of X(t), X(t) we obtain (3.15). Finally, by Lemma 3.1(i) we know that 5^(0 < 0 on (a, a] and 5^(0 > 0 on [a, b), since a matrix and its pseudoinverse have the same definiteness properties. Therefore, the inequalities in (3.16) follow from (3.14) and (3.15). The proof is complete. □ Remark 3.3. It is obvious to see that if the conjoined basis (X, U) has constant kernel on (a,b], then the statements in Lemma 3.2 hold also with a = b. Similarly, if (X, U) has constant kernel on [a, b), then the statements in Lemma 3.2 hold also with a = a. For completeness we note that the conjoined basis (X, U) in Lemma 3.2 is not uniquely determined by (X, U) in the function U. On the other hand, the solution (XP, UP) of (H) is uniquely determined by (X, U), as we showed in [29, Remark 4.5(h)]. Remark 3.4. In order to measure to what extent the controllability assumption on system (H) is violated, the following quantity was introduced in [25, Section 3], see also [29, Section 5] or [31, Section 3]. For fixed a, ft e [a,b] with a < ft we denote by A [a, ft] the linear space of piecewise continuously differentiable vector-valued functions u : [a, ft] M.n, which satisfy the equations u' = —AT(t) u and B(t) u = 0 on [a, ft]. The functions u e A[a, ft] correspond to 922 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 the solutions (x = 0, u) of system (H) on [a, ft]. The space A[a, ft] is finite-dimensional with d[a, ft] := dim A [a, ft] < n. The number d[a, ft] is called the order of abnormality of system (H) on the interval [a, ft]. We remark that system (H) is called normal on [a, ft] if d[a, ft] = 0, while it is called identically normal (or completely controllable) on [a, ft] if d(J) = 0 for every nondegenerate subinterval / c [a, ft]. The integer-valued function d[t, ft] is nondecreasing, piecewise constant, and right-continuous on [a, ft]. Similarly, the integer-valued function d[a, t] is nonincreasing, piecewise constant, and left-continuous on [a, b]. This implies that the limits d£ := lim d[a,t]= max d[a,t], 0 0. It is known in [28, Theorem 3.1.2], see also [29, Theorem 5.2], that these constant quantities are related to the rank of the principal solutions (Xa, Ua) and (Xp, Up). More precisely, K&xXa{t) is constant on (a, a + e], KerXp(t) is constant on [ft — e, ft), and rankX^?) = n — d[a, t] = n — d^ for all t e (a, a + e], (3.19) rankl/KO = n -d[t,ft] = n -dp for all t e [ft - e, ft). (3.20) The relations in (3.19) and (3.20) will be utilized in Section 6 when studying the limit properties of the comparative index. 4. Comparative index and continuous Sturmian theory In this section we derive two main equalities involving the comparative index, as defined in (2.3), in the context of the continuous time linear Hamiltonian system (H). We recall the notation (Xa, Ua) and (Xt,, Ub) introduced in (3.1) for the principal solutions of (H) at the points a and b, respectively. Moreover, if (X, U) and (X, U) are conjoined basis of (H), then we set Y := (XT, UT)T and Y := (XT, UT)T, that is, we define for t e [a, b] the 2n x n matrices ™-$D- ™ The central result of this section allows to express the difference between the numbers of left proper focal points in (a, b] for two conjoined bases (X, U) and (X, U) of (H) in terms of the comparative index evaluated at the endpoints of the considered interval. Similarly for the right proper focal points of (X, U) and (X, U) in [a, b) we use the dual comparative index. For this purpose we introduce the notation P. Šepitka, R. Šimon Hilscher / J. Differential Equations 262 (2017) 914-944 923 mi(a, b] := the number of left proper focal points of (X, U) in (a, b], (4.2) niR[a,b) := the number of right proper focal points of (X, U) in [a, b), (4.3) mi(a,b]:= the number of left proper focal points of (X, U) in (a, b], (4.4) mit[a, b) := the number of right proper focal points of (X, U) in [a, b). (4.5) The left and right proper focal points are always counted including their multiplicities. By (1.5) and (1.6) we then have the equalities niL(a,b]= ^ nti{t), mR[a,b) = ^ ntR{t). We note that under (1.1) the above sums are always finite, since the kernel of X(t) is piecewise constant on [a,b] by Proposition 1.1. With the notation in (4.2)-(4.5) we prove the following main result, which implements the comparative index into the Sturmian theory of continuous time linear Hamiltonian systems (H). Theorem 4.1 (Sturmian separation theorem). Assume that (1.1) holds. Then for any conjoined bases (X, U) and (X, U) of (H) we have the equalities mL(a, b] -mL(a,b] = fi(Y(b), Y(b)) - /x(F(a), Y(a)), (4.6) mR[a, b) - mR[a, b) = /x*(F(a), Y(a)) - fi*(Y(b), Y(b)). (4.7) The proof of Theorem 4.1 is presented at the end of this section. The idea is to prove equalities (4.6) and (4.7) first under an additional assumption on the constant kernel of (X, U) and (X, U) on (a, b] or [a, b), and then apply this partial statement to a suitable partition of the interval [a,b]. Before we proceed to this step we derive several auxiliary statements about the comparative index. Proposition 4.2. Let (X, U) and (X, U) be conjoined bases of(H). Then for all t e [a, b] fi(Y(t), Y(t)) - fi(Y(a), Y(a)) = fi(Y(t), Ya(t)) - fi(Y(t), Ya(t)), (4.8) H*{Y(t), Y(t)) - fi*(Y(b), Y(b)) = fi*(Y(t), Yb(t)) - fi*(Y(t), Yb(t)). (4.9) Proof. Let E be the matrix in (2.5), so that Ya(a) = E. Let O(f), 0(f), Ofl(f), $>b(t) be the fundamental matrices of system (H) such that Y(t) = ®(t)E, Y(t) = 0(t)E, ®a(t)E = Ya(t), ®b(t)E = Yb(t), te[a,b]. (4.10) That is, the fundamental matrices O(f), ^(t), ^a(t), &b(t) are symplectic and they are constructed in such a way that the conjoined bases Y(t), Y(t), Ya(t), Yb(t) form their second 2n x n matrix columns, respectively. This can be done by a suitable choice of conjoined bases which complete Y(t), Y(t), Ya(t), Yb(t) to normalized conjoined bases of (H). For example, the first 2n x n column of the matrix O (t) can be given by Y(t) := (-XT(t),-UT(t))T, where (X, U) 924 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 is the conjoined basis from Lemma 3.2 associated with (X, U), or the first 2n x n column of the matrix 0a(t) is given by ya(t) := (XT(t), Ul(t))T, where (Xa, Ua) is the conjoined basis of (H) with the initial conditions Xa(a) = I and Ua(a) = 0. In particular, we can construct 0a(t) and 0b(t) such that they satisfy 0a(a) = I = 0b(b). It then follows by the uniqueness of solutions of system (H) that 0a(t) 0(a) = 0(t) = 0b(t) 0(b), 0a(t)0(a) = 0(t) = 0b(t)0(b), te[a,b]. (4.11) By using (4.10) and (4.11) and applying formula (2.10) in Proposition 2.2 with the symplectic matrices O := 0a(t), Z := 0(a), and Z := 0(a), we then obtain the equality fi(y(t), y(t)) - fi(y(a), y(a)) = fi(oa(t) 0(a) e, 0a(t) 0(a) e) - /x( 0, (5.20) mL(a, b] - inLb(a, b] (=6) n(Y(b), E) - /x(F(a), Yb(a)) (2= -/x(F(a), Yb(a)) < 0. (5.21) Similarly, for the right proper focal points we have mR[a, b) - mRb[a, b) (=7) fi*(Y(a), Yb(a)) - fi*(Y(b), E) (=9) fi*(Y(a), Yb(a)) > 0, (5.22) mR[a, b) - inRa[a, b) (=7) fi*(Y(a), E) - fi*(Y(b), Ya(b)) (=9)-fi*(Y(b), Ya(b)) < 0. (5.23) These calculations show that the first two inequalities in (5.18) and (5.19) are satisfied. The last equalities in (5.18) and (5.19) follow directly from Theorem 5.3. □ Remark 5.7. According to (5.16) in Corollary 5.4, the lower bounds in (5.18) and (5.19) are the same, as well as the upper bounds in (5.18) and (5.19) are the same. Moreover, these lower and upper bounds are independent on the conjoined basis (X, U). Since these bounds are attained for the specific choices of (X, U) := (Xa, Ua) and (X, U) := (Xb, Ub), the inequalities in (5.18) and (5.19) cannot be improved - in the sense that the estimates (5.18) and (5.19) are satisfied for all conjoined bases (X, U) of (H). The results in Theorem 5.6 yield another improvement of the estimates in (5.4). In particular, in contrast with Theorem 5.2 we obtain the estimates, which do not depend on the chosen conjoined bases (X, U) and (X, U). In addition, it allows to compare the numbers of left proper focal points of (X, U) and right proper focal points of (X, U) and vice versa. P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 931 Corollary 5.8 (Sturmian separation theorem). Assume that (1.1) holds. Then for any conjoined bases (X, U) and (X, U) of (H) we have the estimates | mi(a, b] — mi(a, b] | < rankXfl(Z?) = rankX^(a) < n, (5.24) \mR[a,b) — mR[a,b)\ < mnkXb(a) = rankXfl(Z?) < n, (5.25) | mi(a, b] — mR[a, b) | < rankXfl(Z?) = rankX^(a) < n. (5.26) In particular, for one conjoined basis (X, U) of (H) we have | mi(a, b] — mR[a, b) \ = \ rankX(a) — rankXfJ?) | < rankXa(b) = rankX^(a) < n. (5.27) Proof. Inequality (5.24) follows from (5.18) in Theorem 5.6 applied to (X, U) and (X, U). Similarly, (5.25) follows from (5.19) applied to (X, U) and (X, U). Next we apply inequality (5.18) to (X, U) and inequality (5.19) to (X, U) and use Corollary 5.4 to obtain (5.26). Finally, estimate (5.27) follows from Theorem 5.1 and from inequality (5.26) with (X, U) := (X, U). □ In the proof of Theorem 5.6 we derive the exact formulas mL(a, b] = inLa(a, b] + fi(Y(b), Ya(b)), mR[a, b) = inRb[a, b) + /x*(F(a), Yb(a)), (5.28) which show how to calculate the number of left or right proper focal points of an arbitrary conjoined basis (X, U) of (H) as a sum of a quantity which does not depend on (X, U) and the comparative index of (X, U) with (Xa, Ua) at t = b, or the dual comparative index of (X, U) with (Xb, Ub) at t = a. The formulas in (5.28) then highlight the importance of the comparative index in the Sturmian theory of linear Hamiltonian systems (H). Formulas (5.28) are especially important for theoretical investigations about the proper focal points of (X, U). For practical purposes, e.g. in the oscillation theory, it is more convenient to have estimates for the numbers m^(a, b] and mR[a, b), which do not explicitly involve the possible complicated evaluation of the comparative index. In Theorem 5.9 below we present such estimates of m^ (a, b] and mR [a, b). At the same time we show that the universal lower and upper bounds for m^(a, b] and mR[a, b) in Theorem 5.6 can be improved for some particular choice of (X, U). Theorem 5.9 (Sturmian separation theorem). Assume that (1.1) holds. Then for any conjoined basis (X, U) of '(H) we have the inequalities inLa(a,b] +rankirfl(/V) - min{rankXa(b), rankX(&)} 0 depending on the chosen point to. The next result shows how to compute the multiplicities in (1.5) and (1.6) of left and right proper focal points of (X, U) at some point to by a limit involving the comparative index. Theorem 6.3. Assume that (1.1) holds. Let (X, U) be a conjoined basis of(H) and let (Xt, Ut) be the principal solution of(H) at the point t, i.e., (3.1) holds with s = t. Then with the notation in (4.1) we have mL(to) = lim_/x(F(?0), Y (to)), t0e(a,b], (6.8) mR(t0)= lim fi*(Y(t0),Y,(t0)), t0e[a,b). (6.9) P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 935 Proof. Let to e (a,b] be fixed. For (6.8) we apply formula (4.6) on the interval [t,to] with (X, U) := (Xt, Ut) being the principal solution at t, together with the notation (5.10) and (5.13) on [t, to]. First we observe that the principal solution (XtQ, UtQ) of (H) at to has no right proper focal points in [t, to) when t < to is close enough to to, since the kernel of XtQ(s) is constant for s e[t, to) in this case. Therefore, mRtQ[t, to) = 0 for all t < to close enough to to. This implies by (5.16) that also fnu(t, to] = mRt0[t, to) = 0 for all t < to close enough to to. (6.10) By using Yt (t) = E and formula (2.9) we obtain that /x(Y(t), E) = 0 and hence, mL(to) = \im_mL(t,to]i6=0) \im_{mL(t,to]-mLt(t,to]} t->-t0 (4=6) lim \ii(Y{to), Yt (to)) - ii(Y(t), E)) = lim p(Y(t0), % (to)), t->-t0 which proves (6.8). In a similar way, if to e [a, b) is fixed, then the principal solution (XtQ, UtQ) of (H) has no left proper focal points in (to, t] when t > to is close to to, since the kernel of XtQ (s) is constant for s e (to, t] in this case. Therefore, by (5.16) we obtain inRtito, t) = inu0(to,t] = 0 for all t > to close enough to to. (6.11) By using formula (2.9) we then obtain that /x*(Y(t), E) = 0 and hence, mR(to) = lim mR[t0, t) (6=1} lim {mR[t0, t) - mRt[t0, t)} This shows that (6.9) holds and the proof is complete. □ Remark 6.4. Formulas (6.10) and (6.11) in the above proof have the following interpretation. There exists e > 0 such that for all t e [to — e, to) the principal solution (Xt, Ut) of (H) at the point t has nonincreasing kernel on the interval (t, to], and for all t e (to, to + e] the principal solution (Xt, Ut) of (H) at the point t has nondecreasing kernel on the interval [to, t). Therefore, by (5.15) in Theorem 5.3 together with mRtQ[t, to) = 0, Xt(to) = —Xf(t), and (3.20) we obtain rn~Rt[t, to) ^5=\ankXt(to) = rankXfo(f) <"3=°') n — d[t, to], t e [to — e, to). (6.12) Similarly, from (5.14) and (3.19) together with mu0(to, t] = 0 we obtain tnu(to, t] <'5=4') rankXf(fo) = rankXfo(f) ('3=9') n — d[to, t], t e (to, to + e]. (6.13) 936 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 In Theorem 6.3 we established the existence of the left-hand limit at to of the comparative index /x(F(?o), Y (to)) and the right-hand limit at to of the dual comparative index /x*(F(?o), Y (to)). In the next result we show that these comparative indices have limits at the point to also from the opposite sides. It is surprising that the values of these limits then depend on the abnormality of system (H) in the corresponding left or right neighborhood of the point to. We recall from Remark 3.4 the notation d^ for the maximal order of abnormality of system (H) at the point to from the right and left. Theorem 6.5. Assume that (1.1) holds. Let (X, U) be a conjoined basis of(H) and let (Xt, Ut) be the principal solution of (H) at the point t. Then lim fi(Y(t0), Yt (t0)) =n- d+, t0 e [a,b), (6.14) lim fi*(Y(t0), Y (to)) =n-d~, t0 e (a,b], (6.15) where the numbers d^ and d^ are defined in (3.17) and (3.18). Proof. Let to, t e [a, b] be given. Since the Wronskian of (X, U) and (Xt, Ut) is equal to XT (t), it follows from properties (2.6) and (2.7) that n(Y(t0), Y (to)) + fi*(Y(t0), Y (to)) = rankX(f) + mnkXt(t0) - rankX(f0). (6.16) Assume that to,t e [a,b) with to < t. By (6.13) in Remark 6.4 we know that rankXf(?o) = n — d[to, t] when t is sufficiently close to to. By Proposition 1.1 we also know that rankX(s) = rankX(f^) is constant on (to, t] for t close to to. Combining these facts with the limit in (6.9) in Theorem 6.3 we obtain from (6.16) that the limit of ft(Y(to), Yt (to)) as t —>- t£ exists and lim ft(Y(to), Y (to)) = rankX(^) + lim (n - d[to, t]) - rankX(fo) - lim ii*{Y(to), Y (to)) (3=7) rankX(^) - rankX(f0) An- d^ -mR(t0) ^= n-d+. Therefore, equality (6.14) holds. Next we assume that to,t e (a,b] with t < to. Then similarly as above we obtain from (6.16) by using (6.12), (6.8), (3.18), and (1.5) that the limit of /x*(F(?o), Y (to)) &st—^tQ also exists and lim ft*(Y(to), Y (to)) = rankX(^) + lim (n - d[t, to]) - rankX(fo) - lim ft(Y(t0), Y (to)) rankX^Q-) — rankX(fo) + n — 3-] = 4 (located at 7r, t3, 37r, m^[^-,^L)=2 (located at t3, 27r), 2' 2 and the multiplicity of each left and right proper focal point is 1. Since we have in this case rankXfl(^) = 2 = rankX^(|-), the equalities mLb(%, 7f] = 4 = mLfl(f, ?f ] +rankifl(^f), •2' 2 T") = 4 = m/u,[§, ^) + rankifo(§), m*fl[f, ^) = 4 = mLba, mLfl(f, ^] = 2 = m^[f, ?f) illustrate the validity of Theorem 5.3 and Corollary 5.4. By Theorem 5.6 we conclude that for every conjoined basis (X, U) of (H) we have 2) is W = 2nl. Therefore, by (2.4) we have M(t) =2n[I — X^(t)X(t)]. This shows that M(t) is symmetric and rankA4(?) = def X(t). This implies that V(t) = Af(?) X(t) = P(t) and V(t) = 2jtX\t) X(t) P(t). Since the matrices X(t) and X(t) are diagonal, they commute and indV(t) = indX(t) X(t). Therefore, by (2.3) we have in this case the expressions fi(Y(t), Y(t)) =def X(t) +indA(0 X(t), n*(Y(t), Y(t)) = def X(t) + ind[-A(?) X(t)]. This yields after straightforward calculations that t G [§, Ti) U [T2, T3) U [T4, T5) U [T6, ^), te[n, r2) U [t3, t4) U [t5, r6) U {^}, f G{|}U(T1,T2]U(t3,t4]U(T5,T6], f G (§, Tj] U (T2, T3] U (T4, T5] U (T6, We can see that /x(?) and /x*(?) are piecewise constant on [§,3-] and that /x(?) is right-continuous on [j, lj-) and /x*(?) is left-continuous on (j, ^-], as we claim in Theorem 6.1. The discontinuity points of /x(?) are located at r\-re and at and the discontinuity points of /x*(f) are located at ^ and at t1-t6. Moreover, according to (6.1) and (6.2) the jumps in the values of /x(?) and /x*(?) at each discontinuity point to satisfy /x(?0) - lim ii(t) = mL(t0) - mL{to), /x*(?0) - lim /x*(?) = mR(t0) - mR{to). (7.3) t^TtQ l^l0 fi(t):=fi(Y(t),Y(t)) = fi*(t):=fi*(Y(t),Y(t)) = 0, '2, 942 P. Sepitka, R. Simon Hilscher / J. Differential Equations 262 (2017) 914-944 For example, the differences in (7.3) are equal to 1 when to g {r2, t4, te} and they are equal to — 1 when to e {r\, t3, t5}. Also, Theorem 4.1 is illustrated by the equalities mL(f, \\ - mL(f, f ] = -1 = /x(^) - /x(f), m*[f, ^) - m*[f, ^) = 1 = /x*(f) - /x*(^). Next we will illustrate the validity of Theorems 6.3 and 6.5 at some given to, e.g., at the point to = x2 g [tt, 2it]. For this purpose we need to calculate the value of the principal solution (Xt, Ut) at the point r2 for t in some neighborhood of r2. We can directly verify that (AW.ftW) = ((J sin(s°_,)).(i cos(s°_,)))- MS (7.4) The conjoined basis (X, U) is invertible (and hence has constant kernel) on the intervals [it, r2) and (x2,2it] and it has a left and right proper focal point at r2, with rankX(t2) = 1. Therefore, we obtain from Remark 6.7 that /x*(F(t2),Ff(t2)) = l, /x(F(t2),Ff(t2))=ranki)(t2), te(r2,2n], (7.5) /x(F(t2),Ff(t2)) = l, /x*(F(t2),Ff(t2))=rankif(t2), te[n,x2). (7.6) From (7.4) we get that mnkXt(r2) = 1 for all t g [n, 2n], t ^ t2. Since t2] = 1 for t g *T2 [77-, t2) and J[t2, f] = 1 for t e (%2,2it], it follows by (3.17) and (3.18) that = 1. Then we obtain from (7.5) and (7.6) that lim /x(F(t2), % (t2)) = 1 = mL(r2), lim /x(F(t2), % (r2)) = 1 = n - d+, lim /x*(F(t2), Ff (t2)) = 1 = n - d", lim /x*(F(t2), Ff (t2)) = 1 = mR(r2). These one-sided limits coincide with those in Theorems 6.3 and 6.5, compare also with Remark 6.6. Remark 7.3. The results in this paper (in particular, in Theorems 4.1, 5.2, 5.6, 5.9, 6.1, 6.3 and in Corollaries 5.8, 5.10) are new even for a completely controllable system (H). In this case the left and right proper focal points of (X, U) coincide, since the matrix X(t) is invertible on [a, b] except at finitely many isolated points. Remark 7.4. Some results in this paper about the numbers of focal points of conjoined bases of linear Hamiltonian system (H) can be regarded as continuous analogs of discrete time results for conjoined bases of a symplectic difference system. More precisely, formulas (4.6) and (4.7) correspond to [13, Corollary 3.1 and Equation (3.4)], formula (5.1) corresponds to [13, Equation (3.5)], formulas (5.14) and (5.15) correspond to [7, Equations (4.27a)-(4.27b)], formula (5.16) corresponds to [13, Lemma 3.3] and [7, Equations (4.29a)-(4.29b)], inequality (5.31) corresponds to [3, Theorem 3.2], and formulas (6.8) and (6.9) correspond to [13, Corollary 3.1]. In this respect the discrete time theory motivates the development in the continuous time theory, which is the opposite direction compared to the traditional approach. On the other hand, it is easy P. Šepitka, R. Šimon Hilscher / J. Differential Equations 262 (2017) 914-944 943 to see that the new estimates in Theorems 5.6 and 5.9 and in Corollaries 5.8 and 5.10 can now be derived also in the discrete time setting for symplectic systems. Remark 7.5. During the preparation of the final version of this paper in June 2016, we were notified about the paper [15] by J. Elyseeva. Her paper deals with comparison and separation theorems for left proper focal points in (a,b] of two conjoined bases of two different linear Hamiltonian systems of the form (H) satisfying the Sturm majorant condition. Hence, in some sense her results are more general than ours in this paper. More specifically, formula (4.6) in our Theorem 4.1, the statement (i) in our Theorem 6.1, and equality (6.8) in our Theorem 6.3 are contained in [15] as Theorems 2.2, 2.3, and Lemma 3.1. We would like to emphasize that the results in the present paper were derived independently of [15] and, to our knowledge, in about the same time. Also, our proofs use different techniques and they are more straightforward, as we deal with one system (H) only. In addition, we derive our results for the right proper focal points as well, and relate them with those for the left proper focal points. Remark 7.6. The results in this paper open new directions in the oscillation theory of linear Hamiltonian systems as well as self-adjoint Sturm-Liouville differential equations. For example, singular Sturmian type theorems for a controllable system (H) on unbounded intervals [a, oo), (—oo, b], or (—oo, oo) are proven in [11]. We are convinced that it is now possible to generalize and complete the results in [11] to uncontrollable systems (H) by using the comparative index. We also believe that the comparative index will lead to solving the fundamental questions about possible distribution and locations of (left and right proper) focal points of conjoined bases of (H) in [a, b]. These topics will be discussed in our subsequent work. References [1] A. Ben-Israel, T.N.E. Greville, Generalized Inverses: Theory and Applications, second edition, Springer-Verlag, New York, NY, 2003. [2] D.S. Bernstein, Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory, Princeton University Press, Princeton, 2005. [3] M. Bohner, O. Došlý, W. Kratz, Sturmian and spectral theory for discrete symplectic systems, Trans. Amer. Math. Soc. 361 (6) (2009) 3109-3123. [4] SL. Campbell, CD. Meyer, Generalized Inverses of Linear Transformations, Classics in Applied Mathematics, vol. 56, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2009. Reprint of the 1991 corrected reprint of the 1979 original. [5] WA. Coppel, Disconjugacy, Lecture Notes in Mathematics, vol. 220, Springer-Verlag, Berlin-New York, 1971. [6] O. Došlý, Focal points and recessive solutions of discrete symplectic systems, Appl. Math. Comput. 243 (2014) 963-968. [7] O. Došlý, J. Elyseeva, Singular comparison theorems for discrete symplectic systems, J. Difference Equ. Appl. 20 (8) (2014) 1268-1288. [8] O. Došlý, J. Elyseeva, Discrete oscillation theorems and weighted focal points for Hamiltonian difference systems with nonlinear dependence on a spectral parameter, Appl. Math. Lett. 43 (2015) 114-119. [9] O. Došlý, J. Elyseeva, An oscillation criterion for discrete trigonometric systems, J. Difference Equ. Appl. 21 (12) (2015) 1256-1276. [10] O. Došlý, W. Kratz, A remark on focal points of recessive solutions of discrete symplectic systems, J. Math. Anal. Appl. 363 (1) (2010) 209-213. [11] O. Došlý, W. Kratz, Singular Sturmian theory for linear Hamiltonian differential systems, Appl. Math. Lett. 26 (12) (2013) 1187-1191. [12] JV. Elyseeva, The comparative index for conjoined bases of symplectic difference systems, in: S. Elaydi, J. Cush-ing, R. Lasser, A. Ruffing, V. Papageorgiou, W. Van Assche (Eds.), Difference Equations, Special Functions, and Orthogonal Polynomials, Proceedings of the International Conference, Munich, 2005, World Scientific, London, 2007, pp. 168-177. 944 P. Šepitka, R. Šimon Hilscher / J. Differential Equations 262 (2017) 914-944 [13] J.V. Elyseeva, Comparative index for solutions of symplectic difference systems, Differ. Equ. 45 (3) (2009) 445^-59; translated from Differ. Uravn. 45 (3) (2009) 431^144. [14] J.V. Elyseeva, Comparison theorems for symplectic systems of difference equations, Differ. Equ. 46 (9) (2010) 1339-1352; translated from Differ. Uravn. 46 (9) (2010) 1329-1342. [15] J.V. Elyseeva, Comparison theorems for conjoined bases of linear Hamiltonian differential systems and the comparative index, J. Math. Anal. Appl. 444 (2) (2016) 1260-1273. [16] J.V. Elyseeva, A. Bondarenko, The Schur complement in an algorithm for calculation of focal points of conjoined bases of symplectic difference systems, Int. J. Pure Appl. Math. 67 (4) (2011) 455^-74. [17] P. Hartman, Ordinary Differential Equations, John Wiley, New York, 1964. [18] R. Fabbri, R. Johnson, S. Novo, C. Nunez, Some remarks concerning weakly disconjugate linear Hamiltonian systems, J. Math. Anal. Appl. 380 (2) (2011) 853-864. [19] R. Johnson, S. Novo, C. Núňez, R. Obaya, Uniform weak disconjugacy and principal solutions for linear Hamiltonian systems, in: Proceedings of the International Conference on Delay Differential and Difference Equations and Applications, Balatonfuered, Hungary, 2013, Springer, Berlin, 2014. [20] R. Johnson, S. Novo, C. Núňez, R. Obaya, Nonautonomous linear-quadratic dissipative control processes without uniform null controllability, J. Dynam. Differential Equations (2016), http://dx.doi.org/10.1007/sl0884-015-9495-1, in press. [21] R. Johnson, C. Núňez, R. Obaya, Dynamical methods for linear Hamiltonian systems with applications to control processes, J. Dynam. Differential Equations 25 (3) (2013) 679-713. [22] W. Kratz, Quadratic Functionals in Variational Analysis and Control Theory, Akademie Verlag, Berlin, 1995. [23] W. Kratz, Definiteness of quadratic functionals, Analysis 23 (2) (2003) 163-183. [24] W. Kratz, R. Simon Hilscher, Rayleigh principle for linear Hamiltonian systems without controllability, ESAIM Control Optim. Calc. Var. 18 (2) (2012) 501-519. [25] W.T. Reid, Principal solutions of nonoscillatory linear differential systems, J. Math. Anal. Appl. 9 (1964) 397^-23. [26] W.T. Reid, Ordinary Differential Equations, John Wiley & Sons, Inc., New York-London-Sydney, 1971. [27] W.T. Reid, Sturmian Theory for Ordinary Differential Equations, Springer-Verlag, New York-Berlin-Heidelberg, 1980. [28] P. Sepitka, Theory of Principal Solutions at Infinity for Linear Hamiltonian Systems, PhD dissertation, Masaryk University, Brno, 2014. [29] P. Sepitka, R. Simon Hilscher, Minimal principal solution at infinity for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations 26 (1) (2014) 57-91. [30] P. Sepitka, R. Simon Hilscher, Principal solutions at infinity of given ranks for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations 27 (1) (2015) 137-175. [31] P. Sepitka, R. Simon Hilscher, Principal and antiprincipal solutions at infinity of linear Hamiltonian systems, J. Differential Equations 259 (9) (2015) 4651^1682. [32] P. Sepitka, R. Simon Hilscher, Genera of conjoined bases of linear Hamiltonian systems and limit characterization of principal solutions at infinity, J. Differential Equations 260 (8) (2016) 6581-6603. [33] P. Sepitka, R. Simon Hilscher, Reid's construction of minimal principal solution at infinity for linear Hamiltonian systems, in: S. Pinelas, Z. Došlá, O. Došlý, P.E. Kloeden (Eds.), Differential and Difference Equations with Applications, Proceedings of the International Conference on Differential & Difference Equations and Applications, Amadora, 2015, in: Springer Proceedings in Mathematics & Statistics, vol. 164, Springer, Berlin, 2016, pp. 359-369. [34] R. Simon Hilscher, Sturmian theory for linear Hamiltonian systems without controllability, Math. Nachr. 284 (7) (2011) 831-843. [35] R. Simon Hilscher, On general Sturmian theory for abnormal linear Hamiltonian systems, in: W. Feng, Z. Feng, M. Grasselli, A. Ibragimov, X. Lu, S. Siegmund, J. Voigt (Eds.), Dynamical Systems, Differential Equations and Applications, Proceedings of the 8th AIMS Conference on Dynamical Systems, Differential Equations and Applications, Dresden, 2010, in: Discrete Contin. Dyn. Syst. Suppf, vol. 2011, American Institute of Mathematical Sciences (AIMS), Springfield, MO, 2011, pp. 684-691. [36] M. Wahrheit, Eigenvalue problems and oscillation of linear Hamiltonian systems, Int. J. Difference Equ. 2 (2) (2007) 221-244. APPENDIX D Paper by Sepitka & Simon Hilscher (JDE 2019) This paper entitled "Singular Sturmian separation theorems on unbounded intervals for linear Hamiltonian systems" appeared in the Journal of Differential Equations, 266 (2019), no. 11, 7481-7524, see item [86] in the bibliography. 143 ® ELSEVIER Available online at www.sciencedirect.com ScienceDirect J. Differential Equations 266 (2019) 7481-7524 Journal of Differential Equations www.elsevier.com/locate/jde Singular Sturmian separation theorems on unbounded intervals for linear Hamiltonian systems Peter Šepitka, Roman Šimon Hilscher * Department of Mathematics and Statistics, Faculty of Science, Masaryk University, Kotlářská 2, CZ-61137Brno, Czech Republic Received 12 March 2018; revised 6 September 2018 Available online 7 December 2018 Abstract In this paper we develop new fundamental results in the Sturmian theory for nonoscillatory linear Hamiltonian systems on an unbounded interval. We introduce a new concept of a multiplicity of a focal point at infinity for conjoined bases and, based on this notion, we prove singular Sturmian separation theorems on an unbounded interval. The main results are formulated in terms of the (minimal) principal solutions at both endpoints of the considered interval, and include exact formulas as well as optimal estimates for the numbers of proper focal points of one or two conjoined bases. As a natural tool we use the comparative index, which was recently implemented into the theory of linear Hamiltonian systems by the authors and independently by J. Elyseeva. Throughout the paper we do not assume any controllability condition on the system. Our results turn out to be new even in the completely controllable case. ©2018 Elsevier Inc. All rights reserved. MSC: 34C10 Keywords: Sturmian separation theorem; Linear Hamiltonian system; Proper focal point; Minimal principal solution; Antiprincipal solution; Comparative index This research was supported by the Czech Science Foundation under grant GA16-00611S. Corresponding author. E-mail addresses: sepitkap@math.muni.cz (P. Sepitka), hilscher@math.muni.cz (R. Simon Hilscher). https://doi.Org/10.1016/j.jde.2018.12.007 0022-0396/© 2018 Elsevier Inc. All rights reserved. 7482 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 1. Introduction Let n e N be a given dimension. In this paper we consider the linear Hamiltonian system x' = A(t)x + B{t)u, u' = C(t)x - A7\t)u, t eX, (H) where X c R is a fixed interval (not necessarily compact) and A, B, C : X —>- R"x" are given piecewise continuous matrix-valued functions on X such that B(t) and C(t) are symmetric and the Legendre condition holds, i.e., B(t) >0 forallfeZ. (1.1) The purpose of the paper is to develop new fundamental results in the Sturmian theory of a nonoscillatory system (H) on the interval X = [a, oo) or X = (—oo, b], in particular to derive the Sturmian separation theorems concerning the numbers of focal points in X of conjoined bases of (H). We refer to Section 2 for the definitions of a conjoined basis and the nonoscillation of (H). We show that the known Sturmian separation theorems on a compact interval X = [a, b] can be extended to the unbounded interval X = [a, oo) or X = (—oo, b]. The main ingredients in this extension are the new concept of a multiplicity of a focal point at infinity for conjoined bases of (H), which we introduce in this paper, and using the minimal principal solution of (H) at infinity from [25] as the reference solution for counting the focal points. As a natural tool, which connects these two concepts, we use the comparative index from [9,10], which was recently implemented into the theory of linear Hamiltonian systems by the authors in [30] and independently by Elyseeva in [12,13]. We note that the first applications of the comparative index in the continuous time theory were derived in [11, Section 3]. It is known in [19, Theorem 3] or in [14, Proof of Lemma 3.6(a)] that under (1.1) every conjoined basis (X, U) of (H) has the kernel of X(t) piecewise constant on X, i.e., the kernel of X (t) changes finitely many times in any compact subinterval of X. In this case we say that (X, U) has a left proper focal point sito^X if Ker X(t^) ^ KerX(fo) with the multiplicity mL(t0) := defX(t0) - def X(t~), (1.2) and a right proper focal point at to e X if KerX(f^) ^ KerX(fo) with the multiplicity mR(t0) := def X(t0) - defX(t+), (1.3) see [20,34]. The notations KerX(f^), defX(f^), and later rankX(f^) represent the onesided limits at to of the piecewise constant quantities KerX(f), def X(t) := dimKerX(f), and rankX(f). In the historical development of the Sturmian theory for system (H) on a compact interval X = [a, b] the principal solutions at the points a and b play a fundamental role, see [21, Corollary 1, p. 336], [20, Corollary 4.8], [32, Theorems 1.4-1.5], and recently [30, Theorem 5.6]. We recall that the principal solution (Xs, Us) of (H) at the point s e [a, b] is defined as the solution of (H) starting with the initial conditions Xs(s) = 0, Us(s) = I. (1.4) P. Šepitka, R. Šimon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7483 For future reference in the paper we state the following result from [30, Theorem 5.6]. We emphasize that the focal points are always counted including their multiplicities. A more detailed statement is presented in Proposition 2.2. Proposition 1.1. Assume that (1.1) holds with X = [a,b]. If the principal solution (Xa, Ua) of (H) has m left proper focal points in (a, b], then any other conjoined basis of (H) has at least m and at most m + rank Xa(b) left proper focal points in (a, b]. Similarly, if the principal solution (Xb, Ub) of(H) has m right proper focal points in [a, b), then any other conjoined basis of(H) has at least m and at most m + rank Xb(a) right proper focal points in [a, b). We note that rankXa (b) = rankXb(a), since this quantity is equal to the rank of the (constant) Wronskian of the two solutions (Xa, Ua) and (Xb, Ub)- The above result in Proposition 1.1 holds for a general system (H) without any controllability assumption. Recall that system (H) is completely controllable on X if the trivial solution (x(t), u(t)) = (0,0) is the only solution of (H), for which i(i)=0ona nondegenerate subinter-val of X. When X = [a, oo) resp. X = (—oo, oo) and system (H) is completely controllable on X, then Došlý and Kratz proved in [8, Theorem 1 and Corollary 1] the following results. Proposition 1.2. Assume that (1.1) holds with X = [a, oo), system (H) is completely controllable on [a, oo) and nonoscillatory at oo. If the principal solution of (H) at infinity has m focal points in [a, oo), then any other conjoined basis of(H) has at least m focal points in [a, oo). Proposition 1.3. Assume that (1.1) holds with X = (—oo, oo), system (H) is completely controllable on (—oo, oo) and nonoscillatory at ±oo. Then the principal solutions of(H) at infinity and minus infinity have the same number of focal points in (—oo, oo). For a completely controllable system (H) on [a, oo) we know by [18, Theorem 4.1.3] that Ker X(íq) = {0} for every point toe [a, oo). This means that the notions of left and right proper focal points in (1.2) and (1.3) coincide, i.e., mi(to) = m^(fo), and the corresponding multiplicity is m(řo):=defA(ř0). (1.5) Recall that the principal solution (Aoo, Uoo) of (H) at infinity is defined as a conjoined basis of (H), for which Aoo(f) is invertible on some interval [a, oo) and řhm ( J A"1 (s) B(s) Xl~\s) ds^j = 0. a According to [21, Theorem VII.3.3] or [5, Theorem 3, p. 43] or [16, Section XI.10.5(i)-(ii)], this solution exists and is unique up to a constant right nonsingular multiple when system (H) is nonoscillatory and completely controllable on [a, oo). However, the questions regarding the validity of the estimates in Proposition 1.1 on unbounded intervals (i.e., for b = oo), as well as removing the complete controllability assumption in Propositions 1.2 and 1.3 remained open. In this paper we provide a solution to both of the two above problems. We show that in the absence of the controllability assumption the minimal principal solution of (H) at infinity from 7484 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 [25,26] should be used as the reference solution for counting the (left and right proper) focal points. According to [25, Definition 7.1], this solution is defined as a conjoined basis (Xoo, Uoo) of (H), for which the kernel of Xoo(t) is constant on some interval [a, oo) and t t mn (jxl(s) B(s)xU (s)ds^J =0, (1.6) a where the dagger denotes the Moore-Penrose pseudoinverse [2-4]. Similarly to the controllable case, the minimal principal solution of (H) at infinity exists and is unique up to a constant right nonsingular multiple when the system (H) is nonoscillatory, see [25, Theorems 7.2 and 7.6] and Proposition 2.9. The second main ingredient of the present paper is concerned with the definition of the multiplicity of a focal point at infinity (Definition 3.1) for conjoined bases of (H). This is a completely new notion in the theory of differential equations (it is new even in the controllable case), which is related to a unified view on the principal solutions of (H) at a finite point and at infinity. In [31, Theorem 5.8] we proved that these two types of solutions coincide, that is, the principal solution at to is in fact the (left and right) minimal principal solution of (H) at to in a sense parallel to (1.6). Then, motivated by the result in [31, Theorem 6.1], we define the multiplicity of a focal point of (X, U) at infinity as the difference of the defect of its associated T-matrix 7^ ^ and the order of abnormality of system (H) on [a, oo). The matrix 7^ ^ is defined by t rffi0O := lim Sj(t), Sa(t) := f Xf(s) B(s) XfT (s)ds, te[a,oo), (1.7) a where a e [a, oo) is such that the kernel of X(t) is constant on [a, oo). Using this new concept we prove (Theorem 5.7) that the results in Proposition 1.1 extend naturally to unbounded intervals, when the multiplicities of the left proper focal points are counted in the interval (a, oo]. In particular, the multiplicity of the (left) focal point at infinity should be included. The results in Proposition 1.1 are essentially based on using the comparative index, which was introduced by Elyseeva in [9,10], to express the difference of the numbers of left proper focal points of two conjoined bases (X, U) and (X, U) of (H) in (a,b]. Similarly, the dual comparative index is used for the difference of the numbers of right proper focal points of (X, U) and (X, U) in [a, b), see Subsection 2.2. As a main tool for proving our new Sturmian separation theorems we derive (Theorem 5.1) extensions of the above mentioned formulas to unbounded intervals (a, oo] or [a, oo). In this approach we also apply the limit characterization of the minimal principal solution of (H) at infinity in different genera of conjoined bases from [26, Corollary 5.5], as well as a newly derived characterization of antiprincipal solutions of (H) at infinity in different genera of conjoined bases (Theorems A.l and A.4 in the appendix). Our new results also include optimal estimates for numbers of focal points of one or two conjoined bases (Theorems 5.7 and 5.10 and Corollaries 5.8 and 5.13), as well as limit properties of the comparative index at infinity (Theorem 6.1) and its relationship with the multiplicities of focal points at infinity (Theorem 6.4). As an application of the main Sturmian separation theorems for system (H) we derive a singular version of the Sturmian separation theorem for the second order Sturm-Liouville differential equations (Remark 7.1) and discuss the corresponding notion of disconjugacy on the unbounded interval [a, oo] (Theorem 7.2). We note that all the presented results extend naturally P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7485 to the unbounded intervals of the form (—00, 00], [—00, 00), or (—00, 00) (Remark 8.1). We also wish to emphasize that all the results are new even for the completely controllable system (H). In conclusion, we believe that this paper provides a new perspective in understanding the Sturmian theory of linear Hamiltonian systems and Sturm-Liouville differential equations on unbounded intervals. We are also convinced that these results will stimulate further development in the oscillation theory of differential equations in general. 2. Conjoined bases and their properties In this section we present an overview of the properties of conjoined bases of (H) in the general possibly uncontrollable case. We also recall the definition of a comparative index, the order of abnormality of (H), the nonoscillation and genera of conjoined bases of (H), and the principal and antiprincipal solutions of (H) at infinity. 2.1. Conjoined bases We adopt a usual convention that 2n x n matrix-valued solutions of (H) will be denoted by the capital letters, typically by Y, Y, Y, Y, etc. In this case we split the solutions into two n x n blocks denoted by X and U (preserving the notation in Y), i.e., ™-$0- *»-$?>) for generic conjoined bases of (H), or tM = (*M), r„<,) = (*<°m). F^w = (*-»<") (2.2) \UAt)J VCooW/ \U-oo(t)l for the (minimal) principal solutions of (H) at the point s el, resp. at plus/minus infinity. A solution Y of (H) is a conjoined basis if XT (t) U(t) is symmetric and rankF(f) = n for some (and hence for any) tel. The principal solution Ys for s el, which is given by the initial conditions (1.4), is an example of such a conjoined basis. For two solutions Y and Y of (H) their Wronskian W(Y, Y) := YT(t) J Y(t) = XT(t) U(t) - UT(?) X(t) (2.3) is constant on 1, since its derivative is zero throughout 1. Any conjoined basis Y forms a one half of a symplectic fundamental matrix O(f) of (H), i.e., O(0 = (F(0 Y(t)), Or(0JO(0=J, J:=(_°7 J)- (Z4) In this case the conjoined bases Y and Y are normalized in a sense that their Wronskian is W(Y, Y) = I. This can be equivalently formulated as XUT - XUT = I, XXT = XXT, UUT = UUT, (2.5) 7486 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 saying that O 1 (?) = — J qV (?) J and 0(?) O 1 (?) = /. We note that for any conjoined bases Y, Y, Y of (H) such that W(Y, Y) = I the n x n matrix W(Y, Y) [W(Y, Y)]T is symmetric. (2.6) The proof of (2.6) follows from the properties in (2.5). Next we consider constant real 2n x n matrices F, i e {1,2, 3,4}, and derive additional properties regarding their Wronskian type matrices W(F,-, Yj) = YjjYj. In applications of these properties in Section 4 the matrices F will be the values of conjoined bases of (H) at some fixed point ?o e T. Proposition 2.1. Let Y\, Y2, F3, F4 be real constant 2n x n matrices. Then W(Y1,Y2)W(Y3,Y4)-W(YuY3)W(Y2,Y4) = Y[J(Y2 F3) J (F2 Y3fjY4. (2.7) /nparticular, ifW(Y2, F2) = 0 = W^Fs, F3) and WXFs, F2) = /, then W(YU F2) W(Y3, F4) - W(F1? F3) W(F2, F4) = W(F1? F4). (2.8) Proof. Identity (2.7) follows by direct calculations by using that F2Ff - F3F2r = yjyT with the 2n x 2« matrix y := (F2 F3). If in addition W(F2, F2) = 0 = ^(Fc,, F3) and W(F3, F2) = /, then the matrix y satisfies y = -J, and hence also yjyT = -J. Therefore, identity (2.8) follows from (2.7). □ If F is a conjoined basis of (H), then we use for simplicity the terminology kernel of F, image of F, and rank of Y for the quantities KerX, Iml, and rankX, respectively. In this context the property that F(?) has a constant kernel on some interval Xq<^X means that the kernel of x(t) is constant onXo. 2.2. Focal points and comparative index on compact interval For a conjoined basis F of (H) the multiplicities of its left and right proper focal points are defined, under (1.1), by formulas (1.2) and (1.3). For the interval X = [a, b] we denote by mi(a, b] := the number of left proper focal points of F in (a, b], (2.9) mR[a,b) := the number of right proper focal points of F in [a, b). (2.10) In the same spirit as in (2.9) and (2.10) we will use the notations mi(a,b], mR[a,b) and m,LS(a, b], ntRS[a, b) for the numbers of left and right proper focal points of a conjoined basis F and of the principal solution Ys in the given interval, typically with s £ [a, b, ?} or later in Section 5 with s = ±00. The following regular Sturmian separation theorems were derived in [30, Section 5], compare also with Proposition 1.1. Proposition 2.2. Assume that (1.1) holds with I =[a, b]. Then for any conjoined basis Y of(H) and the principal solutions Ya and Yb we have P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7487 mL(a,b] + rank X(b) = mR[a,b) + rankX(a), (2.11) mLb(a,b] = mLa(a,b]+rsinkXa(b), inRa[a, b) = mRb [a, b) + rank Xb (a), (2.12) mRa[a,b) = mLb(a,b], mLa(a, b] = mRb[a, b), (2.13) m~La(a,b]E, SE) - [i(S 0E, SE) = [i(S S &E) - /x(E, SE) - fi*(S 0E, SE) = fi*(S E, S&E) - (i*(&E, &E). (2.27) In addition, by using (2.20)-(2.22) we can check easily the identities fi(-Y,Y) = fi(Y,-Y) = fi(Y,Y), fi*(-Y,Y) = fi*(Y,-Y) = fi*(Y,Y). (2.28) 2.3. Order of abnormality We denote by d[t, oo) the dimension of the space of vector solutions (x = 0, u) of (H) on the interval [t, oo). This number is called an order of abnormality of system (H) on [t, oo). Then 0 < d[t, oo) < n and the integer-valued function d[t, oo) is nondecreasing, piecewise constant, and right-continuous on [a, oo). Therefore, there exists the maximal order of abnormality d^oi which satisfies doo:= lim d[t, oo) = max d[t,oo), 0 where Y\ and Y2 are conjoined bases of(H) satisfying (2.5) and (2.35) with regard to the conjoined bases Y\ and Y2. If ImXi(a) = 1111X2(0;), then (i) MT N\ and AfJN2 are symmetric and N\ + A J = 0, (ii) M\ and M2 are nonsingular, M\ M2 = M2 M\ = I, and P2M2 = (P\M\)\ (iii) ImAq c ImPi and!mN2 c ImP2. Moreover, the matrices M\, N\ do not depend on the choice of Y\, and the matrices M2, N2 do not depend on the choice of Y2, namely Ni = W(Yi,Y2), N2 = W(Y2,Y1) = -N[, M1 = -W(Y1,Y2), M2 =-W(Y2,Yi). (2.38) The first equality in (2.36) applied to the conjoined bases Y\ and Y2 allows to rewrite expressions (2.37) into the form X3-i(t) = Xi(t)[PiMi + Sia(t)Ni], re [a, 00), ie{l,2}, (2.39) where S\a(t) and (0 are associated with Y\ and Y2 through (1.7). Hence, under the assumptions of Proposition 2.5 we have \mX\(f) = ImX2(r) on [a, 00), that is, the conjoined bases Y\ and Y2 have eventually the same image. 2.5. Genera of conjoined bases Let system (H) be nonoscillatory at 00. As in [26, Definition 6.3 and Remark 6.4] we define a genus G°° of conjoined bases of (H) as an equivalence class of all conjoined bases of (H) which have eventually the same image. In this case we define the rank of G°° as the eventual rank of some (or any) conjoined basis Y e see also [27, Remark 6.4]. Moreover, from (2.32) and (2.29) it follows that n - doo < rank£°° < n. Conjoined bases Y of (H), which have eventually the smallest possible rank n — doo according to (2.32), form the unique minimal genus G^°n- Similarly, conjoined bases having the largest possible rank n form the unique maximal genus G^ax- That is, for Y e G^ax tne matrix X(t) is eventually invertible, see also [26, Remarks 7.14 and 7.15]. In [28, Section 4] we introduced an ordering G^° ^ G^ between two genera of conjoined bases by the inclusion between the images of their representing conjoined bases, i.e., eventually ImXi(f) c ImX2(0 holds for Y\ e G^° and Y2 e Gf- In particular, the results in [28, Theorem 4.8 and Remark 4.7] say that the set T00 of all genera of conjoined bases of (H) forms a complete lattice, where G)°^G^ and G^vG^ denote the genera represented by conjoined bases having their image eventually equal to ImXi(f) n ImX2(r) and ImX\(t) + ImX2(t), respectively. In this case the two genera G^m and G^ax are the smallest and the largest elements of the set T 00 in this ordering. In a similar way we treat the genera of conjoined bases in the neighborhood of — 00 by using the notation G'00, Gj?, G~™, etc. P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7491 Remark 2.6. The theory of genera of conjoined bases of (H) at ±00 discussed above as well as in [24,26-28] extends under the Legendre condition (1.1) in a straightforward way to the left and right neighborhoods of any finite point to e [a, 00). This is a consequence of the fact that the left and right proper focal points of any conjoined basis of (H) are isolated, by [19, Theorem 3]. In this context we will use the notation Q(t^) for the genus corresponding to a conjoined basis F in a left/right neighborhood of the point to, or the notation Qmm{tQ) and Qmdix{to) for the corresponding minimal and maximal genus. This idea is similar to the unification of the theory of principal and antiprincipal solutions at ±00 and at a finite point to, which was recently developed in [31]. 2.6. Relation being contained for conjoined bases In the following subsection we recall a construction of conjoined bases of (H) with constant kernel through a relation "being contained", see [25, Section 5] for more details. Let Y and F* be two conjoined bases of (H) such that Y has constant kernel on [a, 00). Let P and Psaoo be the associated orthogonal projectors for Y defined in (2.31) and (2.33). We say that F* is contained in Y on [a, 00), or that Y contains the conjoined basis F* on [a, 00), if there exists an orthogonal projector P* such that X*(t) = X(t) P* on [a, 00) and Im PSa00 Q Im P* c Im P. (2.40) It follows that every conjoined basis F* of (H), which is contained in a conjoined basis F of (H) with constant kernel on [a, 00), has also a constant kernel on [a, 00) with KerX*(t) = KerP*, X*(t) = X*(t) R*(t) on [a, 00), (2.41) where the matrix R*(t) is defined in (2.31) through X*(t). The importance of the relation being contained can be seen from the following results, see [25, Section 5]. Remark 2.7. (i) In [25, Theorem 5.11] we proved that the relation "being contained" preserves the corresponding S-matrices, and hence also the T-matrices. More precisely, if F is a conjoined basis of (H) with constant kernel on [a, 00) and Sa(t) is the associated matrix in (1.7), then for any conjoined basis F* of (H), which is contained in F on [a, 00), its corresponding matrix S*a (t) satisfies the equality S*a(t) = Sa(t) for all t e [a, 00), and consequently also T*a, 00 = Ta, oo- (ii) From [25, Remark 5.13] it follows that every conjoined basis of (H) from the minimal genus which has constant kernel on [a, 00), can be constructed from a given conjoined basis F with constant kernel on [a, 00) by using the relation "being contained" with the choice of P* := Psaoo- Moreover, as we comment in [26, Remark 7.14] and in Subsection 2.5, for any two conjoined bases Y\ and F2 of (H) belonging to Gm\n there exists a el such that Y\ and F2 have constant kernel on [a, 00) and the equality ImX\(t) = Im X2(t) holds on [a, 00). In particular, if M\ and N\ are the associated constant matrices in Proposition 2.5, then the matrices T\a,oo and T^co in (1-7) corresponding to Y\ and F2 satisfy = M{ T^oo Mi + M(Ni. (2.42) The following result from [28, Remark 4.12] shows that the relation being contained allows to construct conjoined bases from a genus H°° by using conjoined bases from a given genus 7492 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 Q°° satisfying TL°° < Q°°. This construction will be utilized in the proof of Theorem A.4 in the appendix. Proposition 2.8. Assume that (1.1) holds with I = [a, oo). Let Y be a conjoined basis of (H) from a given genus Q°° and let a el be such that Y has constant kernel on [a, oo). Moreover, let P and Psaoo be the associated matrices in (2.31) and (2.33). Then for every genus fi00 < Q°° there exists a unique orthogonal projector P* satisfying (2.40) such that Y contains a conjoined basis of (H) on [a, oo) with respect to P*, which belongs to the genus fi00. 2.7. Principal and antiprincipal solutions at infinity Following the discussion about the S-matrices in (1.7) and Subsection 2.4, we observe that for a conjoined basis Y of (H) with constant kernel on [a, oo) the matrix-valued function S^(t) is nonnegative definite and nonincreasing on [a, oo). Therefore, the matrix Tat00 defined in (1.7) exists, it is symmetric and nonnegative definite, and ImTa00 c Im.P_oo:= lim sUt), 7>_oo<0, d-00 < d(-oo, ft] + rankTg < n, (2.44) compare with (2.43) and with [31, Section 5]. 3. Multiplicity of focal point at infinity In this section we introduce one of the key concepts of this paper, which is the multiplicity of the focal point at 00 for a conjoined basis Y of (H). This notion is motivated by the result in [31, Theorem 6.1 and Remark 6.4], in which we characterized the multiplicity of the left proper focal point at to in (1.2) in terms of the order of abnormality of (H) near to and the rank of the associated T-matrix. More precisely, if Y is a conjoined basis of (H) with constant kernel on the interval [a, to), then mL(to) = n- d[a, t0) - rankTa r, Ta t- := lim_ s\(t), while if Y has constant kernel on the interval (to, a], then ntR(to) = n- d(t0, a] - makTa t+, Ta t+ := lim sl(t), '0 '0 + where d[a, to) and d(to, a] are the orders of abnormality of (H) on the intervals [a, to) and (to, a], respectively. Furthermore, we derive an equivalent formula for the multiplicity at 00 resembling the original definition in (1.2). The results in this section are fundamental for the development of the Sturmian theory of system (H) on the unbounded interval [a, 00) in Section 5. We note that the presented notion and the results are new even for a completely controllable system (H), see Remark 3.7 below. Definition 3.1 (Multiplicity of focal point at 00). Let 1 = [a, 00) and let Y be a conjoined basis of (H) with constant kernel on the interval [a, 00) for some a e [a, 00). We say that Y has a (left) proper focal point at 00 if d[a, 00) + rank Taj00 and Ta^oo in (2.31), (2.33), and (1.7). Let Yoo be the minimal principal solution of(H) at oo. Then Im[W(Yoo, Y)f =Imrffi0O ©Im(P - PSa00), (3.2) rank7^ oo =rankWr(F00, Y) + n -rank£°° -d[a, oo). (3.3) Moreover, the multiplicity of the focal point ofY at oo defined in (3.1) satisfies mi(oo) = rankC?00 — rank W(Yoo, Y). (3.4) Proof. Let a be as in the theorem and choose ft e [a, oo) such that d[f3, oo) = doo - Denote by Tp, oo and Ps^oo me matrices in (1.7) and (2.33) associated with the conjoined basis Y on the interval [ft, oo). Without lost of generality we may assume that the minimal principal solution Foo has constant kernel on [ft, oo). Moreover, let Yp be a conjoined basis of (H) satisfying (2.5) and (2.35) with Y := Foo and a := ft. It then follows from Proposition 2.11 that Yp is a maximal antiprincipal solution of (H) at oo and the equality lim X71(t)Xoo(t)=0 (3.5) t^oo p holds. Following the notation in Proposition 2.5 we represent Y in terms of Yoo and Yp via (constant) matrices Moo and /Voo := W^Foo* Y). That is, mo\_/*„w ,£[(3,0O). (3.6) In particular, the representation in (3.6) implies the formula P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7495 X(t)=X00(t)M00 + Xp(t)N00, te[ft,oo). (3.7) In turn, by combining (3.5) and (3.7) we obtain that lim X~\t) X{t) y=> lim [X~\t) Xoo(0 + Noo- (3.8) -^oo r t^oo r On the other hand, from Corollary A.6 in the appendix with a := ft, Y := Yp, Y := Y, P := P, PSaoo := pSpoo, and fa, oo := Tp, oo it follows that lim X~\t)X(t) = L with lmLT =lmTa oo^lmiP - PSfiO0). (3.9) f^oo ^ p Therefore, by using (3.8) and (3.9) we get the equality Noo = L and Im^=Im7>,00©Im(P-P 0 for every t e [a, 00). Moreover, ifY^ is the minimal principal solution of (H) at 00, then W(Y00,Y)[W(Y00,Y)]T >0. (4.1) Proof. By [30, Lemma 3.2] we know that X(t)XT(t) > 0 on [a,b] for every b e (a, 00), which yields the first part of the theorem. From (2.6) with Y := Foo we know that the matrix WXFoo, Y) [W^Foo, Y)]T is symmetric. Next, choose ft e [a, 00) such that d[ft, 00) = doo and the conjoined bases Y, Y, and F^ have constant kernel on [ft, 00). Let Yp be a conjoined basis of (H) satisfying (2.5) and (2.35) with Y := Foo and a := ft. By Proposition 2.11 we know that Yp is a maximal antiprincipal solution of (H) at 00, while from formula (3.8) in the proof of Theorem 3.3 it follows that W(Foo, F) = lim X-\t)X(t), W(Foo, Y) = lim Xl\t)X(t). (4.2) Consequently, with the aid of (4.2) we obtain Woo, F) [W(Foo, Y)f = lim X~\t) [X(t) XT(t)]XT-\t) > 0. This shows that (4.1) holds. □ 7498 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 In the sequel we will use a symplectic fundamental matrix of nonoscillatory system (H) at oo, which is determined by the minimal principal solution Yoo. According to (2.4) we denote &oo(0 := (Yoo(t) Foo(0), te[a, oo), (4.3) where Y^ is a conjoined basis of (H) which is normalized with Y^, i.e., W(Foo> ioo) = I - Every conjoined basis Y of (H) can be then uniquely represented in the spirit of Proposition 2.5 via the fundamental matrix Ooo(0 and a constant 2n x n matrix Dqq, that is, r(0 = *oo(0A». tela,oo), D0Q = (-W^Y)\, JD0Q = (W^Y)\, (4.4) where the matrix J is given in (2.4) and = O"1^) Y(t) = - J O^(f) JY(t), see (2.38). The following result is a fundamental tool for the proof of the Sturmian separation theorem (Theorem 5.1) in the next section. It is formulated in terms of the matrix JDqq from (4.4). Theorem 4.2. Assume that (1.1) holds with X = [a, oo) and system (H) is nonoscillatory at oo. Let Foo be the minimal principal solution of(H) at oo with the associated matrix Ooo(0 in (4.3). Moreover, let Y be a conjoined basis of (H) with constant kernel on [a, oo) c X belonging to a genus Q°° and let Ya be the principal solution of (H) at the point a. Then /x(j£>co, J^)=rank^oo-rankW(F0O,F), fi*(Y(a), Y^a)) = 0, (4.5) where and are the constant matrices in (4.4) corresponding to Y and Ya. Proof. Following (4.4), let the matrices Dqq and D1^ be split into the n x n blocks as = ("A = (-1?°°$). k== ( ■ (4.6) By (4.4) with £>oo and in (4.6) and by the definition of the comparative index in (2.21)-(2.22) with Y := JDvo and Y := J£>^ (note that (2.20) holds) we get /x(j£>oo, JD^) = rankA4 + indP, (4.7) A4 = (/ - Noo) W, V = VWT'N^N^V, V = I-MfM, (4.8) where W := W(jDoo, JD^). We will show that W = W(Y, Ya). According to (2.3)-(2.4) and (4.4) we obtain for t e 1 that W(Y, Ya) (=3) YT(t) J Ya(t) = Z)£ [6^(0 J doo(0] (=4) Z)£ JD^ P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7499 In particular, evaluating the Wronskian W(Y, Ya) at t = a and using (1.4) with s := a yields the equality W = XT (a). Consequently, the first and the second identity in (4.8) then read as M = (I -NlN00)XT(a), V=VX(a)NlN^V. (4.9) Let P, Psaoo> and 7^ oo be the matrices in (2.31), (2.33), and (1.7), which correspond to Y. In accordance with Theorem 3.3 we have Im^ = Im?^ oo © Im(P — 7>5q,00), while (2.34) implies Im7^ oo clmP. This yields that ImA^clmP, i.e., PN^Noo = N^N^ = N^NooP. (4.10) And since P = [X^(a) X(a)]T by (2.31), the first equality in (4.9) yields that ImA4 = Im(7 - N^Noo) P = lm(P - A^Aoo), (4.11) rank.M = rankP - rank Aoo = rank£°° - rank Aoo- (4.12) Next we will prove that the matrix V satisfies ImIr(«)V = ImiV^, i.e., XT(a)V = N^K (4.13) for some invertible matrix K. Since MV = 0, it follows that N^Noo XT(a) V = XT(a) V by (4.9) and hence, ImXT(a) V c ImN^Noo = ImN^. Conversely, assume that v e ImN^. Then we also have v elmP = ImXT (a) and there exists iceR" such that v = XT (a) w. Then we write XT{a) Vw = XT{a)w - XT(a) M^Mw = v - XT {a) Jvft Mw. But by using (4.9) we have Mw = (I - NooNoo) XT(a) w = (I - N^Noo) v = 0,so that v = XT(a) Vw. Therefore, v e ImXT(a) V and (4.13) is proven. Let Y be a conjoined basis of (H) satisfying (2.5) and (2.35) and set Noo := W(Yoo, Y). Since W(Y, Y)=0 = W(Y, Y) and W(Y, Y) = I, according to formula (2.8) in Proposition 2.1 with Yi := Yoo, Y2 := Y, Y3 := Y, and Y4 := Ya we obtain W(Yoo, Y) W(Y, Ya) - W(Yoo, Y) W(Y, Ya) = W(Yoo, Ya). (4.14) Following the above notation and using the facts that W(Y, Ya) = XT(a), W(Y, Ya) = XT(a), and = W(Yoo, Ya) = X^a), identity (4.14) then has the form NooXT(a)- NooXT(a) = N^=Xl,(a). (4.15) Combining the formula for V in (4.9) with equalities (4.13) and (4.15) and with the identities X(a) = X(a) P, NooN^ = NooN^, and PXT(a) = 0 then implies that V (=9) VX(a) NlN^ V ( =5) VX(a) [Noo XT(a) - Noo XT (a)] V (4.13)^(4.10) ktx^x^jj^t^ k _ vx(a) nI^px7^) V = KTNooNlNooNl, K = ^NooNl, K>0, 7500 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 where the last inequality follows from (4.1) in Lemma 4.1. Therefore, indV = 0. Upon combining (4.7) and (4.12) we get fi(jDoo, JD^) = mnkM = rank£°° - rankA^. This shows the first formula in (4.5). For the second formula in (4.5) we have by (2.21)-(2.22) with 7 := Y(a) and 71=700(0;) fi*(Y(a), Y00(a))=mnkM + md(-V), (4.16) M = (I - P)W, V = VWTXf(a)X0Q(a)V, V = I-MfM, (4.17) where W = W(Y, 7^) = -N^ by (4.6). According to (4.10) we have M = 0, so that V = I and V = —Noo X^(a) X00(a) by (4.17). Consequently, using the value X 00(a) in (4.15) we get -V = Noo X\a) [Noo XT(a) - Noo XT(a)]T = A^PA^ - Aoo X\a) X(a) A^ where the last inequality follows from Lemma 4.1. Therefore, ind(—V) = 0 and hence, equation (4.16) yields that [i*(Y(a), Yoo(a)) = 0. The proof is complete. □ Remark 4.3. The second formula in (4.5) follows also from the algebraic properties of the comparative index in [10, Property 3, p. 448] and from Lemma 4.1. The first property in (4.5) can be then obtained by (2.24) and [10, Property 3, p. 448] again as follows: r(JDoo, JD^) (2= } rankX(o<) - fi(jD^, J£>co) = rank£°° - fi(jO^(a) E, JO"1 (a) oo, JDoo) - fi(Y(a), Y(a)), (4.20) mR[a, oo) -mR[a,oo) = fi*(Y(a), Y(a)) - /x*(j£>oo, JDoo), (4.21) where Doo and Doo are the constant matrices in (4.4) corresponding to Y and Y. Proof. We define the constant symplectic matrix S := -JO^(o;), where Ooo(0 is given in (4.3), and consider the symplectic fundamental matrices O(0 and O(0 of (H) such that O(f) E = Y(t) and O(0 E = Y(t) on [a, oo), with the constant 2n x n matrix E from (2.25). In addition, let be the matrix in (4.4) corresponding to the principal solution Ya of (H) at the point a. Then we have Y(t) = Ooo(0 Doo, Y(t) = Ooo(0 Doo, and Ya(t) = 0oo(t) on [a, oo). It follows that for t = a we have JDoo = S<$>(a)E, JDoo = S 0(a) E, JDaOQ = -SE. (4.22) By Corollary 4.4 applied to Y and Y together with transformation formula (2.26) with the matrices S := —S, O := 0(a), and O := 0(a) we obtain mL(a, oo] - mL(a, oo] (4= } /x(j£>oo, JD%\) - /x(j£>oo, JD%\) (4=2) ^(-5 0(a) E, -SE) - /i(-S$(a) E, -SE) (2=6) ix(-S 0(a) E, -S0(a) E) - p(0(a) E, 0(a) E) (4=2) h(JDoo, JDoo) ~ li{Y(a), Y(a)), which shows formula (4.20). Next, by (4.3) we know that Ooo(0 JE = Yoo(0 on [a, oo). Thus, Y00(a)=S~1E, Y(a) = S~1S0(a)E, Y(a) = S~lS 0(a) E. (4.23) Combining Corollary 4.4 applied to Y and Y and transformation formula (2.27) with the matrices S := S~l, 0:=S 0(a), and O := S 0(a) then yields mR[a, oo) - mR[a, oo) (4=9) /x*(Y(a), Yoo(a)) — p*(Y(a), Yoo(a)) (4=3) ix*(S~1S0(a) E, S^E) - ix*(S~1S0(a) E, S'1 E) (2=?) ii*(S~lS 0(a) E, S~lS 0(a) E) - fi*(S 0(a) E, S 0(a) E) (4.23)^(4.22) ^^(a), Y(a)) - /X*(-JDoo, - JDoo) (2^8) fi*(Y(a), Y(a)) - /x*(j£>oo, JDoo), which shows formula (4.21). The proof is complete. □ 7502 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 5. Singular Sturmian separation theorems In this section we derive Sturmian separation theorems for the numbers of left proper focal points, resp. right proper focal points, of two conjoined bases of a nonoscillatory system (H) on the unbounded intervals (a, oo] or (—00, b], resp. [a, 00) or [—00, b). The results regarding left proper focal points include the multiplicities of focal points at 00 as we discussed in Section 3, while the results regarding right proper focal points include the multiplicities of focal points at — 00. In addition, we will also derive the corresponding results for the open intervals (a, 00) or (—00, b). As in the previous sections we do not impose any controllability assumption and the results are new even for a completely controllable system (H). The following result corresponds to formulas (2.15) and (2.16) in the case of a compact interval X = [a, b], see [30, Theorem 4.1] and Remark 5.2 below. Theorem 5.1 (Singular Sturmian separation theorem). Assume that (1.1) holds with I = [a, 00) and system (H) is nonoscillatory at 00. Then for any conjoined bases Y and Y of (H) we have the equalities mL(a, 00] - mL(a, 00] = ix{JDoo, JDoo) - /x(F(a), Y(a)), (5.1) mR[a, 00) - mR[a, 00) = fi*(Y(a), Y(a)) - fi*(jDoo, JD^), (5.2) where and are the constant matrices in (4.4) corresponding to Y and Y. Proof. Let a e T be such that both conjoined bases Y and Y have constant kernel on the interval [a, 00). Applying formulas (2.15)-(2.16) on the interval [a, a] we obtain that ffli(a, a] — m^(a, a] = ii(Y{a), F(a)) — /x(F(a), F(a)), (5.3) mR[a,a) -mR[a,a) =/x*(F(a), Y(a)) -/x*(F(a), Y(a)). (5.4) On the other hand, according to (4.20)-(4.21) in Lemma 4.5 we have the identities mL(a, 00] -mL(a, 00] = ^(jDoo, JDoo) - /x(F(a), Y(a)), (5.5) mR[a, 00) - mR[a, 00) = /x*(F(a), Y(a)) - £i*(jZ>oo, JDoo). (5.6) Since mi{a, 00] = mi{a, a\-\-mi{a, 00] and rhi,(a, 00] = rhi,(a, a] + mL(a, 00], adding equalities (5.3) and (5.5) yields the formula in (5.1). Similarly, from mR[a,oo) = mR[a,a) + mR[a, 00) and mR[a, 00) = mR[a,a) + mR[a, 00) together with (5.4) and (5.6) we obtain the formula in (5.2). The proof is complete. □ Remark 5.2. The results in Theorem 5.1 represent the true "singular version" of the formulas in (2.15) and (2.16), which deal with a compact interval T = [a, b]. Indeed, for a fixed point s e [a, b] we consider the (symplectic) fundamental matrix s(s) = — J. Then for every conjoined basis Y of (H) there exists a unique constant 2n x n representation matrix Ds such that, in spirit of Proposition 2.5, Using (5.7) and (5.8) with s := a and s := b yields that Y(a) = -JDa and Y(b) = -JDb. Similarly, for another conjoined basis Y of (H) we have Y(a) = —JDa and Y(b) = —JDb. Therefore, by using property (2.28) of the comparative index, the formulas (2.15) and (2.16) can be rewritten as mL(a, b] -mL(a,b] = fi(JDb, JDb) - fi(JDa, JDa), mR[a, b) -mR[a,b) = v*(JDa, JDa) - ^*(JDb, JDb), while the formulas (5.1) and (5.2) in Theorem 5.1 can be rewritten as mL(a, oo] - mL(a, oo] = ^(JD^, JZ>oo) - p(JDa, JDa), mR[a, oo) -mR[a,oo) = ix*(JDa, JDa) - /AJAxd, J^oo)- By considering a special choice of the conjoined basis Y in Theorem 5.1 we obtain the following formulas. They highlight the importance of the minimal principal solution Yoo of (H) at oo in counting the exact number of left and right proper focal points of any conjoined basis Y of (H) in the intervals (a, oo] and [a, oo). Also, they correspond to formulas (2.17) and (2.18) in the case of a compact interval 1 = [a, b], see [30, Equation (5.28)]. Corollary 5.3. Assume that (1.1) holds with I = [a, oo) and system (H) is nonoscillatory at oo. Then for any conjoined basis Y of(R) we have the equalities mL(a, oo] = mLa(a, oo] + ^(jDoo, JD^), (5.9) mR[a, oo) = inRoo[a, oo) + /x*(F(a), Foo(«))» (5.10) where and D1^ are the constant matrices in (4.4) corresponding to Y and Ya. Proof. Formula (5.9) follows from (5.1) with Y := Ya, since in this case Doo = with the notation used in (4.4) and (4.6), and /x(F(a), Ya(a)) = /x(F(a), E) = 0 by (2.25). Similarly, formula (5.10) follows from (5.2) with Y := Foo, since in this case Doo = (I, 0)T and hence, /x*(J£>oo, jboo) = iJL*(JDoo,-E) = 0. □ In the next statement we connect the multiplicities of left and right proper focal points of one conjoined basis Y of (H) in an unbounded interval. This result corresponds to formula (2.11), see also [30, Theorem 5.1]. 7504 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 Theorem 5.4. Assume that (1.1) holds with X = [a, oo) and system (H) is nonoscillatory at oo. Let Too be the minimal principal solution of (H) at oo. Then for any conjoined basis Y of (H) its numbers of left proper focal points in the interval (a, oo] and right proper focal points in the interval [a, oo) satisfy mi(a, oo] +rankWr(F00, Y) = mR[a, oo) + rankX(a). (5.11) Proof. Let Y be a fixed conjoined basis of (H). According to Proposition 2.4 we choose a >a such that Y has constant kernel on [a, oo). By (2.11) applied to the interval [a, a] we get mi(a, a] + rank X (a) = mR[a, a) + rank X (a). (5.12) In particular, the equalities m^{a,a] = mi(a,oo), mR[a,a) = mR[a,oo), and rankX(aO = rankC?00 hold, where Q°° is the genus corresponding to Y. Then formula (5.12) reads as mi(a, oo) + rankC?00 = mR[a, oo) + rankX(a). (5.13) Finally, combining formula (3.4) in Theorem 3.3 and equality (5.13) yields mt(a, oo] + rank^(Foo, Y) = m^(a, oo) + mi(oo) + rank W^Foo, Y) mi(a, oo) + rankC?00 <"5=3') mR[a, oo) + rankX(a), which shows identity (5.11). The proof is complete. □ In the remaining results of this section we will use the principal solution Ya of (H) at the point a and the minimal principal solution Too of (H) at oo. For these particular conjoined bases of (H) we have WiY^, Ya) = X^ia) and the statement of Theorem 5.4 yields mLoo(a, oo] = mRoo[a, oo) +rankX00(a), (5.14) mLa(a, oo] + rankXoo(a) = mRa[a, oo). (5.15) In the following statement we relate the numbers of left and right proper focal points of Yoo and Ya in (a, oo] and [a, oo). This result corresponds to formulas (2.12) and (2.13), see also [30, Theorem 5.3, Corollary 5.4]. Theorem 5.5. Assume that (1.1) holds with I =[a, oo) and system (H) is nonoscillatory at oo. Then we have mL00(a,oo, JD%A = [i(-E, JD%A = rankiM(a) by (2.25). Similarly, the second formula in (5.16) follows from (5.10) with Y := Ya, since in this case /x*(Ya(a), Foo(«)) = Ijl*(E, Foo(a)) = rankXoo(a) by (2.25). Finally, the formulas in (5.17) follow directly from (5.15) and (5.16). □ Remark 5.6. The results in Theorem 5.5 yield interesting connections with the limits of the corresponding equalities in (2.13). First of all, it is not at all clear whether the limits lim inLb(a,b], lim inRb[a,b) (5.19) exist, and if they exist, then what are their values. Below we show that both of these limits indeed exist and that the first one is equal to rni^a, oo] as we would formally expect, but surprisingly the second one is not equal to rriRooia, oo) in general. More precisely, we have ^ i,n(2-13) r - r u\ ^ r , (5.17) _ hm niLb(a,b\ = hm mRa[a, b) = mRa[a, oo) = mi00(a,cx)J, b^)-oo b^)-oo lim mRb[a,b) (2=3) lim inLa(a, b] = inLa(a, oo) (5=?) inRoo[a, oo) - mLa(oo). b^oo b^oo The above calculation shows that the second limit in (5.19) is equal to the formally expected value inRoo[a, oo) only when mia(oo) = 0, i.e., only when the principal solution Ya is antiprincipal at oo according to Remark 3.2(h). Equations (5.9) and (5.10) yield the lower bounds niL(a, oo] > inLa(a, oo], mR[a,oo)>mRO0[a,oo) (5.20) for the numbers of left and right proper focal points of any conjoined basis Y of (H) in the interval (a, oo] and [a, oo). Observe that both lower bounds are the same according to (5.17). Observe also, that the second estimate in (5.20) generalizes Proposition 1.2 to possibly uncontrollable system (H). In the next statement we provide the corresponding optimal upper bounds for the numbers m^{a, oo] and niR[a, oo). These estimates correspond to (2.14) in the case of a compact interval X = [a, b], see [30, Theorem 5.6]. Theorem 5.7 (Singular Sturmian separation theorem). Assume that (1.1) holds with X = [a, oo) and system (H) is nonoscillatory at oo. Then for any conjoined basis Y of(H) we have inLa(a, oo] < m^(a, oo] < m,Loo(a, oo], (5.21) fnRoola, oo) - oo in (5.27) we obtain mLa(a, oo) < mL(a, oo) < mLa(a, oo) + rank^°. It remains to prove the last equality in (5.25), for which we utilize the results in Theorems 5.5 and 3.3. In particular, we have inLoo(a, oo] (5=6) inLa(a, oo] + rank(a) =inLa(a, oo) + inLa(oo) + rank(a) (3= mLa(a, oo) + rank- rank W(Y0Q,Ya)+ rankX^(a) = mLa (a, oo) + rank , because W(Foo, Ya) = X^a). Therefore, (5.25) is established. For (5.26) we have by (5.22) with a :=t and by the second equation in (5.16) with a :=t that mRoo[t,oo) - a+ in (5.28) we obtain iriRooia, oo) aa, where aa '■ = inf {a e [a, oo), Ya has constant kernel on the interval [a, oo)}. Let b > aa. With this choice of b the conjoined basis Ya has constant kernel on the interval [b, oo), so that m^a(b, oo) = 0. Then by the lower bound in (5.25) on the interval (b, oo) and with Y := Ya we obtain that mib(b, oo) < m^a(b, oo) = 0. Hence, mib(b, oo) = 0 as well. Then ^ ^ ^ (2 12) ^ mLb(a,oo) = mLb(a,b] + mLb(b,oo) = mLa(a,b]+mnkXa(b) = inLa (a, oo) + rank Qf (5=5) inLoo (a, oo]. (ii) In a similar way we show that the lower and upper bounds in (5.26) are also optimal. In particular, the lower bound in (5.26) is attained for Y := F^, and the upper bound is attained for Y := Yb for b sufficiently close to a, i.e., m,Rb(a, oo) = htRa[a, oo) for b sufficiently close to a. Namely, we will prove that the latter equality holds for all b e (a, ftoo), where ftoo := sup {ft e (a, oo), F^, has constant kernel on the interval (a, ft]}. (5.29) Let us fix b e (a, ftoo)- Then we have by the second formula in (5.17) with a :=b that inRb(a, oo) = inRb(a, b) + mRb[b, oo) (5=?) inRb(a, b) + inLoo(b, oo]. (5.30) Now we show that htRb(a,b) = 0. By (5.29) we know that Yoo has constant kernel on the interval [t, b) for every t e (a, b), i.e., niRooit, b) = 0 for all t e (a, b). Then by (2.14) with a := t and Y := Foo we conclude that m~Rb[t, b) = 0 for all t e (a, b). Upon taking the limit as t a+ we obtain that inRb(a, b) = lim^fl+ inRb[t, b) = 0. Next, since Fqo also satisfies ntL00(a, b] = 0 by (5.29), it follows that WlLoo(b, OO] — W?Loo (a, oo], while the latter quantity is equal to mRa[a, oo) by (5.17). Therefore, it follows from (5.30) that m,Rb(a, oo) = htRa[a, oo). Remark 5.12. (i) Since by Remark 3.2(h) the multiplicity of the focal point of Foo at oo is ^Loo(oo) = n — doo, it follows that frii^a, oo] = m 1,00(0, 00) + n — doo- This means that the wrong upper bound from (5.23) has to be adjusted (i.e., increased) by the correction term n — doo-This shows that this correction term does not depend on the left endpoint a and that the maximal number of left proper focal points of F in (a, 00) or in (a, 00] is always greater or equal to n — doo - Moreover, the upper bound in (5.23) is indeed optimal only when doo = n. In the latter case every conjoined basis of (H) satisfies mi(oo) = 0. Similarly, the wrong upper bound in (5.24) has to be increased to the number mRa[a, 00) by the correction term m,Ra(a) =n — d£, where rf+ is the maximal order of abnormality of (H) in the right neighborhood of a, see [31, Equation (2.17)]. (ii) In the proof of Theorem 5.10 we showed that the upper bound in (5.25) is attained by the principal solution Yb when b > aa, i.e., inib(a, 00) = mioo(a, 00]. This means that also m,Lb(a, 00] = inioo(a, 00], and consequently mib(oo) = 0. In other words, Yb is an antiprincipal P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7509 solution at oo for every b > aa. This observation is in agreement with [27, Proposition 5.15] for large b. (iii) By similar arguments as in part (ii) adjusted to the upper bounds in (5.26) and (5.22) we obtain that for all b e (a, ftoo) the conjoined basis ft, satisfies mRt,(a) = 0. As an analogy of Corollary 5.8 we obtain from Theorem 5.10 the following. Corollary 5.13. Assume that (1.1) holds with I =[a, oo) and system (H) is nonoscillatory at oo. Then for any two conjoined bases Y and Y of(H) we have | mi(a, oo) — mi(a, oo) | < rankC?^0 < n, (5.31) | mR(a, oo) — mR(a, oo) | < rankXoo(a+) < n. (5.32) Proof. The result in (5.31), resp. in (5.32), follows from estimate (5.25), resp. from estimate (5.26), applied to the two conjoined bases Y and Y. □ Our final result in this section connects the multiplicities of left and right proper focal points of the principal solutions Ya and Yqq in the open interval (a, oo). Corollary 5.14. Assume that (1.1) holds with I =[a, oo) and system (H) is nonoscillatory at oo. Then we have the equality mLa(a, oo) + rank^ = inRoo(a, oo) + rankXoo(a+), (5.33) where is the genus of the principal solution Ya near oo. In particular, the equality mLa{a,oo) = mRoo{a,oo) (5.34) holds if and only if rank = rankXoo(a+). Proof. By (3.4) and (1.3) we have mLfl(oo) = rankC - rank W(Foo, Ya) = rank£fl°° - rank 1^(a), mRoo{a) = rank Zoo (a+) -rankX00(a). Therefore, equation (5.33) is equivalent with the first equality in (5.17). The second statement in the corollary then follows directly from (5.33). □ In the completely controllable case every conjoined basis Y of (H) has X(t) invertible near a and near oo. Therefore, the condition rankC^0 = rankX00(a+) = n is automatically satisfied and we get from Corollary 5.14 the following. This result is also new even in this special setting. Corollary 5.15. Assume that (1.1) holds with I = [a, oo) and system (H) is completely controllable on [a, oo) and nonoscillatory at oo. Then the principal solutions Ya and Y^ satisfy the equality in (5.34), i.e., Ya and Y^ have the same number of focal points in (a, oo). 7510 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 Remark 5.16. It is easy to see that under assumption (1.1) with 1 = (—oo, b] and for a nonoscil-latory system (H) at — oo the results in this section hold also for the numbers of left and right proper focal points of the conjoined bases Y, Y-oo, and ft, in the intervals (—oo, b] and [—oo, b), respectively in the open interval (—oo, b). 6. Asymptotic properties of comparative index In this section we apply the results in Section 5 to obtain asymptotic formulas for the comparative indices fx(Y(t), Y(t)) and fx*(Y(t), Y(t)) when t -> ±oo. Moreover, in addition to (3.4) and (3.14) we derive another representation formulas for the multiplicities mi(oo) and mR(—oo) for a conjoined basis Y in terms of limits at ±oo of comparative indices involving Y and the principal solution Yt. These results essentially extend the limit properties of the comparative index in (2.19) or in [30, Section 6] to the case of to = ±oo. Theorem 6.1. Assume that (1.1) holds with I =[a, oo) and system (H) is nonoscillatory at oo. Then for any two conjoined bases Y and Y of (H) the limits of the comparative indices [i(Y(t), Y(t)) and fx*(Y(t), Y{t))fort -> oo exist and /Xoo(F, Y) := mn fi(Y(t), Y(t)) = [i(JD^, JD^) - mL(oo) + mL(oo), (6.1) /x^(Y,F) :=^fi*(Y(t),Y(t)) = fi*(JD00,JD00), (6.2) where Doo and Doo are the constant matrices in (4.4) corresponding to Y and Y. Proof. By Theorem 5.1 we know that for every t e [a, oo) fi(Y(t),Y(t)) = /x(JDoo, J/3oo) -mL(t,oo]+mL(t,oo], (6.3) fi*(Y(t), Y(t)) = /x*(J£>oo, J/3oo) + mR[t, oo) - mR[t, oo). (6.4) Since system (H) is nonoscillatory at oo, the conjoined bases Y and Y have no left and right proper focal points in the intervals (t, oo) and [t, oo) for sufficiently large t, i.e., m^(t, oo) = 0 = mi(t, oo) and mR[t, oo) = 0 = mR[t, oo) for large t. Therefore, upon taking the limit as t oo in (6.3) and (6.4) we obtain (6.1) and (6.2), respectively. □ Remark 6.2. By taking Y := Too we have JDoo = — E and hence, ixiJDoo, JDqo) = 0 by (2.28) and (2.25). Therefore, in this case equation (6.1) with mi(oo) = inloo(°o) = n — doo and equation (6.2) yield (3 1) ti00(Y,Y00)=n-d00-mL(oc) = rankT^oo, /x^(F, F^) = 0, where a e [a, oo) is such that Too has constant kernel on [a, oo) and d[a, oo) = doo. This reveals another interesting property of the matrix Ta, oo associated with Y, namely that its rank is equal to the limit of the comparative index /x(Y(t), Fqo(O) as f ^ oo. P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7511 Remark 6.3. By [30, Theorem 6.1] or [12, Theorem 2.3] we know that the comparative index /x(F(f), Y(t)) is right continuous, the dual comparative index /x*(F(f), Y(t)) is left continuous, and the formula lim fi(Y(t), Y(t)) = fi(Y(t0), Y(t0)) - mL(t0) + mL(t0) holds. Therefore, the left discontinuity of /x(F(f), Y(t)) at to measures the difference between the multiplicities mi(to) and invito). From this point of view the results in (6.1) and (6.2) of Theorem 6.1 can be interpreted as a compactification of these properties on the extended interval [a, oo] = [a, oo) U {oo}, where we would define /x(F(oo), F(oo)) := [i{JDoo, JDoc) and /x*(F(oo), F(oo)) := [i*(JDoo, JDoo). In this case the left discontinuity of fx(Y(t), Y(t)) at oo, i.e., formula (6.1), measures the difference between the multiplicities mi(oo) and inl(oo). In the next theorem we present a formula for calculating the multiplicity of the focal point at oo of a conjoined basis F in terms of the comparative index of F with the principal solution Yt, respectively in terms of their representing matrices Doo and D'^ in (4.4) and (4.6). This result corresponds to [30, Theorems 6.3 and 6.5] in the case of a compact interval 1. Theorem 6.4. Assume that (1.1) holds with I =[a, oo) and system (H) is nonoscillatory at oo. Then for any conjoined basis Y of (H) the limits of the comparative indices ijl(JDoo, JD1^ and li*{JDoc, JDfoc)for t —> oo exist and lim fi(jD>oc, JD'^) = mL(oo), (6.5) lim ix* (JDoc , JD^) =n-doc, (6.6) where Doc and D1^ are the constant matrices in (4.4) and (4.6) corresponding to Y and Yt. Proof. By Corollary 5.3 and Theorem 5.5 we have for every t e [a, oo) that r(JDoc, JD'cc) (5= mL(t, oo] - mLt(t, oo] (5=?) mL(t, oo] - mRoo[t, oo). (6.7) Since system (H) is nonoscillatory at oo, the conjoined bases F and Yoo have no left and right proper focal points in the intervals (t, oo) and [t, oo) for sufficiently large t, i.e., m^(t, oo) = 0 = m~Roo[t, oo) for large t. Therefore, upon taking the limit as t —>- oo in (6.7) we obtain (6.5). Alternatively we can use the first formula in (4.19) (with a := t) to obtain (6.5) directly. Next we show that formula (6.6) follows from (6.5) by using the relationship between the dual comparative index and the comparative index in (2.23) and (2.24). We fix t e [a, oo). By the form of the matrices JD'^ in (5.18) with a :=t and JDoo in (4.4) we obtain from (2.23) that h* (JD^, JDoc) + rank Zoo (0 = r(JDoo, JD^) +rankW(Foo, Y), (6.8) where we also used that W(Yoo, Yt) = X^it). Moreover, from (2.24) we get r*{jDoo, JD^) + ^{JD1^, JDoo) =ranktF(J£>oo, JD^) =rankX(f), (6.9) 7512 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 where we used that W{JDoo, JD1^) = W(Y, Yt) = XT (t). Upon subtracting (6.8) from equation (6.9) we obtain /x*(j£>co, JD'^) = rank Zoo (0 + rankX(f) - rank W(Foo, Y) - /x(j£>oo, JD^). (6.10) If G^°n and G°° are the genera of conjoined bases corresponding to Foo and Y, respectively, then rank Zoo (0 = rank^^n = n — doo and rankX(f) = rankC?00 for large t. This fact together with the validity of (6.5) imply that the limit of //.*(l7D00, JD1^ as t —>- oo exists, and by (6.10) and (6.5) it is equal to lim /x*(JDoo, J£>oo) = n ~ doo + rank£°° - rank W(Foo, Y) - mL(oc) (=4) n-doo- t^oo v ' The proof of formula (6.6) is complete. □ Remark 6.5. If assumption (1.1) holds with X = (—oo,b] and system (H) is nonoscillatory at — oo, then analogous results as in Theorems 6.1 and 6.4 and Remark 6.2 hold for the limits of the corresponding comparative indices as t —>- —oo. More precisely, we have the formulas /x_oo(F, Y) := ( hrn^FX?), Y(t)) = ^(JD-^, JD.^), /x* (F, Y) := lim li*{Y{t),Y{t))=li*{JD-O0,jb-O0)-mR{-oo) + fhR{-oo), t^—oo fi-oo(Y, F-oo) =0, .utooiY, F-oo) =rankr/!i_oo, where ft e (—oo, b] is such that F_oo has constant kernel on (—oo, ft] and d(—oo, ft] = d-oo. Moreover, we have the limits lim /x*(JD_oo, J£>_oo) = mR(-oo), t^-—oo lim ixijD-oo, JDLqo) =n- d-oo, t^-—oo where .D-oo, D-^, and are the constant matrices corresponding to Y, Y, and Yt with respect to the symplectic fundamental matrix 0_oo(t) = (F_oo(0 ^-oo (0) involving the minimal principal solution F_oo of (H) at —oo, i.e., as in (4.4) and (5.8) we have _ / W^F-oo, Y) \ ~ _ W(Y^,Y)\ _ W(Y^,Yt)\ JD-™-[w(Y^,Y))> JD-°°-\w(Y.00,Y))' ^-"-{wif^Y,))- Remark 6.6. The results in Theorems 5.1 and 6.1 and in Remark 6.5 allow to interpret the limit case of the formulas (2.15) and (2.16) for b —^ oo. Indeed, by (5.1) and (6.1), respectively by (5.2) and (6.2), for any conjoined bases F and Y we have P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7513 mi(a, oo) — mi(a, oo) /Xoo(Y, Y)-fi(Y(a),Y(a)), nin[a, oo) — mR\a, oo) ^(Y(a),Y(a))-^00(Y,Y), mi(—oo, b] — mi{—oo, b] fi(Y(b),Y(b))-fi-oo(Y,Y), mn(—oo, b) — mn(—oo, b) nt^Y, Y)-fi*(Y(b),Y(b)). These formulas represent the continuous time versions of the limit formulas for conjoined bases of one discrete symplectic system in [7, Theorem 4.1]. 7. Sturm-Liouville differential equation In this section we will discuss some of the results from Section 5 for the second order Sturm-Liouville differential equation where r, p :[a, oo) —>- M are given piecewise continuous functions such that r(t) > 0 on [a, oo). As it is common for piecewise continuous functions, the assumption of the positivity of r means that the one-sided limits of r(t) at its points of discontinuity are also positive, so that 1/r is also piecewise continuous on [a, oo). Under these assumptions equation (SL) is a special case of system (H) with n = 1, A(t) = 0, B(t) = l/r(t), and C(t) = —p(t), that is, x := y and u :=r(t)y'. For classical results (oscillation, nonoscillation, disconjugacy) about equation (SL) with continuous coefficients on [a, oo) we refer to [5,16,21,22]. We note that equation (SL) or the corresponding system (H) with the above coefficients is completely controllable on [a, oo). In order to apply the results from Section 5, we denote by ya the principal solution of (SL) at a, i.e., it is determined by the initial conditions ya(a) = 0 and y'a(a) = 1/r (a). We recall that the principal solution yoo of a nonoscillatory equation (SL) at oo is defined by the condition fa° VtKO Joo(f)]d? = oo, where a e [a, oo) is such that yoo(t) ^0 on [a, oo). The principal solutions ya and yoo are uniquely determined (up to a nonzero constant multiple). The zeros (i.e., focal points in the terminology of the previous sections) of a nontrivial solution y of (SL) are defined by the condition y(to) = 0, when to is a finite point. In addition, according to (3.15) in Remark 3.7 we say that y has a zero at oo if y = yoo, i.e., if y is principal at oo. All the zeros of y are simple, i.e., their multiplicities are 1. In accordance with (1.5) we will use the unified notation m(to), in (to), iha(to), moo(to) for the multiplicities of to of the solutions y, y, ya, yoo- That is, for a solution y of (SL) we denote m(to) := mi(to) for to e (a, oo] or m(to) := m#(?o) for to e [a, oo), where the equality mi(to) = m#(?o) holds for every to e (a, oo). Similar notation will be used for the numbers of zeros of y, y, ya, yoo in some given interval. In this context m(oo) = 0 if and only if y is an antiprincipal solution at oo, or in other words l/[r(t) y2(t)] dt < oo for some a e [a, oo) such that y(t) ^ 0 on [a, oo). Remark 7.1. Recently we have observed the paper [1], which studies the validity of the singular Sturmian comparison theorem for two equations of the form (SL) under a standard (strict) majorant condition. However, the approach in this reference does not allow to apply the results to one equation (SL), hence to deduce the corresponding singular Sturmian separation theorem. As a consequence of Theorems 5.5 and 5.7 we obtain the corresponding statement as follows. Assume that equation (SL) is nonoscillatory at oo and let y\ and y2 be two linearly independent [r(t)y')' + p(t)y(t) =0, ie[a,oo), (SL) 7514 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 solutions of (SL). If y\ has two consecutive zeros at ?i, ?2 £ [«, °°L h < {2, then y2 has exactly one zero in the open interval (t\, Another motivation for the study of equation (SL) we found in the paper [6], which uses the principal solution y^ to derive some existence theorems for second order nonlinear differential equations. In particular, in [6, Remark 1, Lemma 5] the authors study the question when yoo(t) ^0 on the whole interval [a, oo) (7.1) and relate this condition to the disconjugacy of (SL) on the interval [a, oo). Following the classical terminology, see [16, Section XI.6] or [5], we define equation (SL) to be disconjugate on an interval Xq c [a, oo] if any nontrivial solution of (SL) has at most one zero in Xq. Note that the right endpoint oo is now included in the above definition. It is well known that the disconjugacy of (SL) on [a, oo) is a condition, which is necessary but not sufficient for the validity of (7.1). This fact is illustrated by [6, Example 1], where (SL) is disconjugate on [a, oo) but the principal solution yoo satisfies y~oo(t) > 0 on (a, oo) and yoo(o) = 0, i.e., (7.1) does not hold. This clearly means that yoo = ya (up to a nonzero constant multiple) and this solution has two zeros in the interval [a, oo]. Hence, equation (SL) from [6, Example 1] is not disconjugate on [a, oo] in the present setting. The following characterization of (7.1) is an immediate consequence of the above definition and the results in Theorems 5.5 and 5.7. We can see that the formulation is in the same spirit as in the regular case, see [5, Theorems 1.1 and 1.2]. Theorem 7.2. The following statements are equivalent. (i) Condition (7'.1) holds. (ii) The solution ya is positive on (a, oo) and antiprincipal at oo. (hi) Equation (SL) is disconjugate on the interval [a, oo]. (iv) There exists a solution of (SL) with no zeros in the interval [a, oo]. Proof. Assume that (i) holds, i.e., /Wgofo, oo) = 0. Then by (5.17) we get ma(a, oo] = 0, so that ya(t) 7^ 0 (positive) on (a, oo), as well as mfl(oo) = 0. The latter condition means that ya is not principal at oo, i.e., (ii) holds. Next we assume that (ii) is satisfied, that is, ma(a) = 1 and ma(a, oo] = 0. Then mfl(oo) = 0 and ma[a, oo) = 1. By (5.15) we get rank})00(a) = 1, so that by (5.16) we have m^a, oo] = 1. This implies through (5.21) that m(a, oo] < 1 for every nontrivial solution y of (SL). If equation (SL) has a solution y with two zeros t\ < ?2 in [a, oo], then y is not a constant multiple of ya, and hence a 0 on (a, oo) and joo(0 > 0 on [a, oo). Also, since m00(oo) = 1, the solutions ya and yco are linearly independent. Then the solution y := ya + yoo satisfies y(t) > 0 on [a, oo) and y is not principal at oo, i.e., m(oo) = 0. Hence, y has no zeros in [a, oo]. Finally, if (iv) is satisfied and y is the solution with no zeros in [a, oo], then by (5.22) with y := y we have m^a, oo) < m[a, oo) = 0. This means that yoo has no zeros in [a, oo), i.e., (7.1) holds. The proof is complete. □ P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7515 The results in Remark 7.1 and Theorem 7.2 clearly hold also on the unbounded intervals [—oo, b] or [—oo, oo], see also Remark 8.1, which we illustrate by the following example. Example 7.3. Equation y" = 0 has the principal solution joo = y-oo = 1 at ±oo, which is positive on (—oo, oo) but has two zeros in the interval [—oo, oo] (its zeros are at ±oo). Therefore, this equation is disconjugate on (—oo, oo), (—oo, oo], or [—oo, oo), but it is not disconjugate on the interval [—00,00]. Moreover, according to Remark 7.1 every solution y2, which is linearly independent with y\ = 1, has exactly one zero in the open interval (—00, 00). This obviously holds, since y2(0 = kt + q for k, q e M with k 7^ 0. 8. Conclusions and remarks In this paper we have presented Sturmian separation theorems for possibly uncontrollable linear Hamiltonian systems (H) on unbounded intervals [a, 00) or (—00, b]. We showed that the classical Sturmian separation theorems on a compact interval [a, b] obtained recently in [12,30] can be extended to unbounded intervals by using suitable properties of the comparative index. The key feature is to define properly the multiplicities of proper focal points at 00 and —00 (Definition 3.1, Theorem 3.3, and Remark 3.6) and to count the multiplicities of proper focal points including those at 00 (for the left proper focal points) and at —00 (for the right proper focal points). The main results contain exact formulas for the difference of the numbers of proper focal points of two conjoined bases Y and Y (Theorem 5.1), exact formulas for the numbers of proper focal points of one conjoined basis Y (Corollary 5.3), the relationship between the numbers of left and right proper focal points for a given conjoined basis Y (Theorem 5.4), comparison of the numbers of proper focal points of the (minimal) principal solutions Ya and Yqq (Theorem 5.5), optimal lower and upper bounds for numbers of proper focal points of any conjoined basis Y (Theorems 5.7 and 5.10), optimal estimates for the difference of proper focal points of any two conjoined bases Y and Y (Corollaries 5.8 and 5.13), and asymptotic formulas for the comparative index and the dual comparative index involving two conjoined bases Y and Y (Theorems 6.1 and 6.4). Also, in the appendix below we derived new characterizations of antiprincipal solutions of (H) at 00 in terms of a Wronskian (Theorem A.l) and in terms of a limit (Theorem A.4), which are used in the proof of Theorem 3.3 to calculate the multiplicities of focal points at 00. We emphasize that all the results in this paper are new even for a completely controllable linear Hamiltonian system (H). We are convinced that the contributions in this paper will motivate further development of the oscillation theory for linear Hamiltonian systems and Sturm-Liouville differential equations. For example, unified Sturmian separation theorems on regular intervals (i.e., compact intervals) and on singular type intervals can be obtained as an immediate consequence of the results in [30] and those in this paper. We will provide detailed statements of this unified theory in a separate note. Furthermore, the new notion of a multiplicity of a focal point at ±00 (Definition 3.1 and Remark 3.6) will lead to a new interpretation of other classical topics in the theory of differential equations, such as the disconjugacy of system (H) on unbounded intervals [a, 00), [a, 00], or [—00, 00]. This topic will also be addressed in our subsequent work. Remark 8.1. In this paper we considered linear Hamiltonian system (H) on the unbounded intervals of the form [a, 00) or (—00, b], i.e., for intervals with one singular endpoint. The formulations of the main results discussed above show that the presented theory remains valid also for the interval X with two singular endpoints, i.e., for a = —00 and 1 = M = (—00, 00). This 7516 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 can be seen by splitting the interval 1 into two disjoint intervals (—oo, b] U (a, oo) with b = a and by using that for any conjoined basis F we have mi(—oo, oo] = mi(—oo, b] + m^{a, oo], mR[—cc, oo) = mR[—oo, b) + mR[a, oo). In this case the quantities Y(a), Y(a), YOQ(a) and rankX(a), rankXoofa), rank^^0 should be replaced by the quantities j7T>_oo, JD-oo, JD^oo and rankW^F-oo, F), rankW^F-oo, F30), rankCP0^, respectively, compare with Remarks 5.2 and 5.16. More precisely, for a system (H) which satisfies the Legendre condition (1.1) with X = M and which is nonoscillatory both at oo and at —oo, we have from Theorem 5.1 for any conjoined bases F and F of (H) the equalities mL(-oo, oo] - mL(-oo, oo] = ^(JD^, JD^) - /x(j£>_oo, JD-oo), (8.1) mR[-oo, oo) - mR[-oo, oo) = ix*(JD_oc, JD_oc) - ^(JD^, JD^), (8.2) from Corollary 5.3 for any conjoined basis F of (H) the equalities iL(-oo, oo] = mL-ooC-oo, oo] + /x(j£>oo, J£>-°°), (8.3) i*[-oo, oo) = mRoo[-oo, oo) + /x*(JD_oo, JDf^), (8.4) from Theorem 5.4 for any conjoined basis F of (H) the equality mi mi m^(—oo, oo] + rank W(Foo, F) = mR[—oo, oo) +rankWr(F_00, F), from Theorem 5.5 the equalities mLoo{-oo, oo] = mL_oo(-oo, oo] + rank WXF30, F_oo), iriR-ool-oo, 00) = mRoo[-oo, 00) + rank W(Yoo, F_oo), mL-oo(-oo, 00] = mRoo[-cc, 00), m«_oo[-oo, 00) = mLoo(-oo, 00], from Theorems 5.7 and 5.10 for any conjoined basis F of (H) the optimal estimates iriL-ooi-oc, 00] < mL(—oo, 00] < mLoo(-oo, 00], inRoo{—oq, 00) < m#[—00, 00) < m^_oo[—00, 00), mL_oo(-oo, 00) oo exists and satisfies lim Xf(t) X(t) P* = L with lmLT = Imfa ^ © Im(P* - 7\j ). (A.4) Proof. Let P* be the orthogonal projector in Proposition 2.8 defined through Y and H°° and let Y* and Y* be the conjoined bases of (H) from the genus H°°, which are contained in Y and Y on [a, oo) with respect to P* and P*, respectively. Moreover, let R*(t) and R*(t) be the matrices in (2.31) associated with and Y*. By using the properties of genera of conjoined bases and the relation "being contained" and (2.41) we have the equalities R*(t) = R*(t), X*(t) = X(t) P*, X^(t)=Xf(t)R*(t), te[a,oo). (A.5) In particular, the identities in (A.5) together with R*(t) X*(t) = X*(t) on [a, oo) yield X\t) X(t) (=5) X\t) X*(t) = X\t) R*(t) X*(t) (=5) Xf (0 R*(t) X*(t)(=5) X< (t) X*(t), t G [a, oo). (A.6) In this way we transfer the study of the limit in (A.4) into the genus H°°, where we can apply the known result from [27, Theorem 6.3]. Let Sa(t), Sa(t), S*a(t), S*a(t), and (suppressing the index oo)Ta:=Ta,oo,fa:= fai0Q, T*a := oo,T*a:= 7^ ^ be the corresponding 5-matrices and T-matrices defined in (1.7). According to Remark 2.7(i) we have the equalities 7520 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 Sa(t) = S*a(t), Sa(t) = 5*0,(0, te[a,oo), (A.7) PSaOO = PS*aOO> ^ ^M-1P5tt0o = P5MooM-1, A^* = PsMOOA*PsMoo. (A. 11) Moreover, denoting by T**a := P**^ 00 and T**a := P**^ 00 the corresponding T-matrices defined in (1.7), we obtain by Remark 2.7(i) that T**a = T*a and T**a = T*a. Hence, by (2.42) in Remark 2.7(ii) we get P*<* = P**<* (2=] Ml f^a M« + Ml A** = Ml f*a M« + Ml A**. (A.12) Suppose now that statement (i) holds, i.e., F is an antiprincipal solution of (H) at 00. Then also F* is an antiprincipal solution at 00 by Remark 2.10, and according to [27, Theorem 6.3] (with Q := H°°, (A, U) := F*, and (A, U) := F*) we have that lim A.f(0 A*(0 = L with m\LT = ImfM ©Im(P* - P^ ). (A.13) /—>-00 ^*aOO In turn, with the aid of (A.6) and (A.8) the relation in (A.13) is equivalent with (A.4). This shows the validity of statement (ii). Conversely, assume (ii), i.e., (A.4) holds. Denote by Lq := P*M~l — L, where L is given in (A.4). Then by (A.6) and (A.8) we have that F* and F* satisfy (A.13). Consequently, by (A. 10) P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7521 we get (t) N* —>- Lo as t —> oo. Moreover, the definitions of the matrices Ps„oo and T*a in (2.33) and (1.7) yield Ps„oo = lim 5^(0 S*a(t) = lim 4,(0 x lim S*a(t) = T*a L0. (A.14) This implies that KerLo c Ker (Ps„oo N^). On the other hand, for any v e Ker (Psmoo N^) we have that v e Ker[S*a(t) Nj[] for all sufficiently large t e [a, oo) and hence, v e KerLo-Therefore, KerLo = Ker(Psmoo N%) holds, which is equivalent with ImLT = Im(N* Psmoo)-By combining the latter identity with the last formula in (A. 11) we get l™(PS„co Lo) = ^(PSmoo ^* PS*aco) (A=" JmN». (A.15) In addition, from the definition of Lo and the second identity in (A.l 1) it follows that L° PS„oo = P* M-1 Psn0Q ~ L P^ (Ai1} PS.^ M"1 - L P^. (A.16) Furthermore, since by (A.13) the inclusion ImLr c Imf*a © Im(P* — P$ ) holds, utilizing the equalities P$ T^a = T±a and P$ (A — P$ ) = 0 we obtain that Im(P^ LT) c Imfw, or equivalents Ker T*a c Ker (L P? ). (A.17) O^q? oo O^q? oo Next, we will prove the inclusion Ker 7^ c K&x Psm00. Let v e KerL^q,. The last identity in (A.ll), the symmetry of Psmoo and T*a, and equality (A.14) then yield N„ v (Ai1} PSn0Q N. PSmoo v (A=4) PSta00 LT0 Tw v = 0. (A.18) Consequently, by using (A.12) and (A.18) we get M^f*a M**v = T*a v — M^N^v = 0. And since the matrix M** is nonsingular, we have M** v e Ker T*a and hence, M** v e Ker(LP^ ), by (A.17). On the other hand, formula (A.15) and the symmetry of MT^N** imply that Im(Ml LTQ) = Im(N^ M**), i.e., Ker(L0 M**) = Ker (Af£ N„). And since the equality A\*u = 0 holds, we have M** d e Ker(LoP^ ). Thus, the vector O^q- oo v e Ker (L P^ ) n Ker (L0 P^ ). Moreover, from (A. 16) it then follows that PSwoo v = PS„oo M-1 M** v (A=6) (Lo + L P^J M« v = 0. Therefore, v e Psmoo and the inclusion KerL^ c KerPsm0o is established. This means that Im Ps„oo ^ Im 7*0,. On the other hand, the opposite inclusion Im T*a c Im Ps„oo always holds by (2.34) and hence, we have ImT^ = ImP,5M00 and rankL^q, = n — d[a, oo). Therefore, the conjoined basis is an antiprincipal solution of (H) at oo. In turn, by Remark 2.10 the conjoined basis Y is also an antiprincipal solution of (H) at oo. The proof is complete. □ 7522 P. Sepitka, R. Simon Hilscher / J. Differential Equations 266 (2019) 7481-7524 Remark A.5. The proof of Theorem A.4 implies that the second condition in (A.4) can be equiv-alently replaced by the inclusion ImLT ctafai00©ta(P*-P§a00). (A.19) Indeed, in the proof of (i) =>■ (ii) we showed that (A. 13) holds, which by Ta = T*a from (A.8) implies (A.19). In the opposite direction (ii) =>■ (i) we utilized just the inclusion (A.19). In the last part of this section we present several special cases of Theorem A.4, which are also new in a sense that they extend known statements in the literature. These special cases are concerned with the situations when either G°° is the maximal genus 9%^, or one of the genera H°° of G°° is the minimal genus G^n. These considerations will also show that Theorem A.4 is new even for the completely controllable system (H). First we consider the situation when Y is a maximal conjoined basis of (H) near oo. In this case we obtain from Theorem A.4 the following characterization of maximal antiprincipal solutions of (H) at oo. Corollary A.6. Assume that (1.1) holds with I = [a, oo) and system (H) is nonoscillatory at oo. Let Y be a maximal conjoined basis of (H) near oo and let Y be a conjoined basis of (H). Let a e [a, oo) be such that d[a, oo) = doo, the matrix X(t) is invertible on [a, oo), and Y has constant kernel on [a, oo). Moreover, let Taj00, P, and Pg ^ be the matrices in (1.7), (2.31), and (2.33), which correspond to Y. Then the following statements are equivalent. (i) The conjoined basis Y is a maximal antiprincipal solution of (H) at oo. (ii) The limit ofX~l(t) X(t) as t oo exists and satisfies lim X~l(t) X(t) = L with lmLT = lmfa © Im(P - P* ). (A.20) Proof. The result follows from Theorem A.4 for the choice of G°° := ■ m this case we have U°° = G^A G°° =G°°, and hence = P and X(t) P* = X(t) P = X(t) on [a, oo). □ The other special cases of Theorem A.4 are concerned with the situation when H°° is the minimal genus G^m- ^ne notati°n and context of the following remark refers to Theorem A.4. Remark A.7. (i) If "H00 = G°° A G°° = G™„, then P* = P$ and condition (A.4) reduces to lim Af(0A(f)Pc =L with ImLr = Imfo, (A.21) In this case Theorem A.4 with (A.21) extends [28, Corollary 5.3] to the situation when Y is not necessarily a principal solution of (H) at oo. (ii) If G°° '■= Gmi„, then part (i) of this remark applies with P = P~ and (A.21) reduces to lim Af(f)X(t) = L with ImLr = Imfo, oo- (A.22) P. Šepitka, R. Šimon Hilscher / J. Differential Equations 266 (2019) 7481-7524 7523 In this case Theorem A.4 with (A.22) extends [28, Corollary 5.4] to the situation when F is not necessarily a principal solution of (H) at oo. (hi) If G00 := G™ax and Q00 := Q™m, then part (ii) of this remark holds and (A.22) reduces to lim X~l(t) X(t) = L with lmLT = m\Ťa (A.23) t^oo In this case Theorem A.4 with (A.23) extends [28, Corollary 5.5] to the situation when Y is not necessarily a principal solution of (H) at oo. Note that condition (A.23) is also a special case of (A.20) in Corollary A.6 for the choice of Q00 := Q™n. Finally, we comment separately the situation of a completely controllable system (H). Even in this very special case the result in Theorem A.4 (or Remark A.7(iii) with G^ax = &min) *s new and it reads as follows. We also note that the assumption of the eventual complete controllability of (H) can be replaced by the weaker condition doo = 0 with the same conclusion. The latter condition is a version of a weak controllability condition used in [15, Hypothesis 2.7] or in [17, Condition D2w,p. 260]. Corollary A.8. Assume that (1.1) holds with I = [a, oo) and system (H) is nonoscillatory at oo and eventually completely controllable. Let Y and Y be two conjoined basis of (H). Let a e [a, oo) be such that the matrices X(t) and X(t) are invertible on [a, oo) and let Ta^ ^ be the matrix in (1.7), which corresponds to Y. Then the following statements are equivalent. (i) The conjoined basis Y is an antiprincipal solution of(H) at oo. (ii) The limit ofX~l(t) X(t) as t oo exists and satisfies (A.23). Remark A.9. Following the last parts of Subsections 2.5 and 2.7 we conclude that the results in this section hold without any change for genera of conjoined bases of a nonoscillatory system (H) at —oo, with the orthogonal projectors and limits considered for t —>- —oo. References [1] D. Aharonov, U. Elias, Singular Sturm comparison theorems, J. Math. Anal. Appl. 371 (2) (2010) 759-763. [2] A. Ben-Israel, T.N.E. Greville, Generalized Inverses: Theory and Applications, second edition, Springer-Verlag. New York, NY, 2003. [3] D.S. Bernstein, Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory. Princeton University Press, Princeton, 2005. [4] S.L. Campbell, CD. Meyer, Generalized Inverses of Linear Transformations, Reprint of the 1991 corrected reprint of the 1979 original, Classics in Applied Mathematics, vol. 56, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2009. [5] W.A. Coppel, Disconjugacy, Lecture Notes in Mathematics, vol. 220, Springer-Verlag, Berlin-New York, 1971. [6] Z. Došlá, M. Marini, S. Matucci, A Dirichlet problem on the half-line for nonlinear equations with indefinite weight. Ann. Mat. Pura Appl. (4) 196 (1) (2017) 51-64. [7] O. Došlý, J. Elyseeva, Singular comparison theorems for discrete symplectic systems, J. Difference Equ. Appl. 20 (8) (2014) 1268-1288. [8] O. Došlý, W. Kratz, Singular Sturmian theory for linear Hamiltonian differential systems, Appl. Math. Lett. 26 (12) (2013) 1187-1191. [9] J.V. Elyseeva, The comparative index for conjoined bases of symplectic difference systems, in: S. Elaydi, J. Cush-ing, R. Lasser, A. Ruffing, V. Papageorgiou, W. Van Assche (Eds.), Difference Equations, Special Functions, and Orthogonal Polynomials, Proceedings of the International Conference, Munich, 2005, World Scientific, London. 2007, pp. 168-177. 7524 P. Šepitka, R. Šimon Hilscher / J. Differential Equations 266 (2019) 7481-7524 [10] J.V. Elyseeva, Comparative index for solutions of symplectic difference systems, Differ. Equ. 45 (3) (2009) 445^-59; translated from Differ. Uravn. 45 (3) (2009) 431^144. [11] J.V. Elyseeva, Generalized oscillation theorems for symplectic difference systems with nonlinear dependence on spectral parameter, Appl. Math. Comput. 251 (2015) 92-107. [12] J.V. Elyseeva, Comparison theorems for conjoined bases of linear Hamiltonian differential systems and the comparative index, J. Math. Anal. Appl. 444 (2) (2016) 1260-1273. [13] J.V. Elyseeva, On symplectic transformations of linear Hamiltonian differential systems without normality, Appl. Math. Lett. 68 (2017) 33-39. [14] R. Fabbri, R. Johnson, C. Nunez, On the Yakubovich frequency theorem for linear non-autonomous control processes, Discrete Contin. Dyn. Syst. 9 (3) (2003) 677-704. [15] R. Fabbri, R. Johnson, S. Novo, C. Nunez, Some remarks concerning weakly disconjugate linear Hamiltonian systems, J. Math. Anal. Appl. 380 (2) (2011) 853-864. [16] P. Hartman, Ordinary Differential Equations, John Wiley, New York, 1964. [17] R. Johnson, R. Obaya, S. Novo, C. Núňez, R. Fabbri, Nonautonomous Linear Hamiltonian Systems: Oscillation, Spectral Theory and Control, Developments in Mathematics, vol. 36, Springer, Cham, 2016. [18] W. Kratz, Quadratic Functionals in Variational Analysis and Control Theory, Akademie Verlag, Berlin, 1995. [19] W. Kratz, Definiteness of quadratic functionals, Analysis 23 (2) (2003) 163-183. [20] W. Kratz, R. Simon Hilscher, Rayleigh principle for linear Hamiltonian systems without controllability, ESAIM Control Optim. Calc. Var. 18 (2) (2012) 501-519. [21] W.T. Reid, Ordinary Differential Equations, John Wiley & Sons, Inc., New York-London-Sydney, 1971. [22] C.A. Swanson, Comparison and Oscillation Theory of Linear Differential Equations, Academic Press, New York, 1968. [23] P. Sepitka, Theory of Principal Solutions at Infinity for Linear Hamiltonian Systems, PhD dissertation, Masaryk University, Brno, 2014. [24] P. Sepitka, Riccati equations for linear Hamiltonian systems without controllability condition, Discrete Contin. Dyn. Syst. (2018), in press. [25] P. Sepitka, R. Simon Hilscher, Minimal principal solution at infinity for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations 26 (1) (2014) 57-91. [26] P. Sepitka, R. Simon Hilscher, Principal solutions at infinity of given ranks for nonoscillatory linear Hamiltonian systems, J. Dynam. Differential Equations 27 (1) (2015) 137-175. [27] P. Sepitka, R. Simon Hilscher, Principal and antiprincipal solutions at infinity of linear Hamiltonian systems, J. Differential Equations 259 (9) (2015) 4651^1682. [28] P. Sepitka, R. Simon Hilscher, Genera of conjoined bases of linear Hamiltonian systems and limit characterization of principal solutions at infinity, J. Differential Equations 260 (8) (2016) 6581-6603. [29] P. Sepitka, R. Simon Hilscher, Reid's construction of minimal principal solution at infinity for linear Hamiltonian systems, in: S. Pinelas, Z. Došlá, O. Došlý, P.E. Kloeden (Eds.), Differential and Difference Equations with Applications, Proceedings of the International Conference on Differential & Difference Equations and Applications, Amadora, 2015, in: Springer Proceedings in Mathematics & Statistics, vol. 164, Springer, Berlin, 2016, pp. 359-369. [30] P. Sepitka, R. Simon Hilscher, Comparative index and Sturmian theory for linear Hamiltonian systems, J. Differential Equations 262 (2) (2017) 914-944. [31] P. Sepitka, R. Simon Hilscher, Focal points and principal solutions of linear Hamiltonian systems revisited, J. Differential Equations 264 (9) (2018) 5541-5576. [32] R. Simon Hilscher, Sturmian theory for linear Hamiltonian systems without controllability, Math. Nachr. 284 (7) (2011) 831-843. [33] R. Simon Hilscher, On general Sturmian theory for abnormal linear Hamiltonian systems, in: W. Feng, Z. Feng, M. Grasselli, A. Ibragimov, X. Lu, S. Siegmund, J. Voigt (Eds.), Dynamical Systems, Differential Equations and Applications, Proceedings of the 8th AIMS Conference on Dynamical Systems, Differential Equations and Applications, Dresden, 2010, in: Discrete Contin. Dynam. Systems, Suppl. 2011, American Institute of Mathematical Sciences (AIMS), Springfield, MO, 2011, pp. 684-691. [34] M. Wahrheit, Eigenvalue problems and oscillation of linear Hamiltonian systems, Int. J. Difference Equ. 2 (2) (2007) 221-244. APPENDIX E Paper by Sepitka & Simon Hilscher (JDE 2020) This paper entitled "Singular Sturmian comparison theorems for linear Hamiltonian systems" appeared in the Journal of Differential Equations, 269 (2020), no. 4, 2920-2955, see item [87] in the bibliography. 189 ® Check for updates ELSEVIER Available online at www.sciencedirect.com ScienceDirect I Differential Equations 269 (2020) 2920-2955 Journal of Differential Equations www.elsevier.com/locate/jde Singular Sturmian comparison theorems for linear Hamiltonian systems * Peter Šepitka, Roman Šimon Hilscher * Department of Mathematics and Statistics, Faculty of Science, Masaryk University, Kotlářská 2, CZ-61137Brno, Czech Republic Received 28 June 2019; accepted 12 February 2020 Available online 25 February 2020 Abstract In this paper we prove singular comparison theorems on unbounded intervals for two nonoscillatory linear Hamiltonian systems satisfying the Sturmian majorant condition and the Legendre condition. At the same time we do not impose any controllability condition. The results are phrased in terms of the comparative index and the numbers of proper focal points of the (minimal) principal solutions of these systems at both endpoints of the considered interval. The main idea is based on an application of new transformation theorems for principal and antiprincipal solutions at infinity and on new limit properties of the comparative index involving these solutions. This work generalizes the recently obtained Sturmian separation theorems on unbounded intervals for one system by the authors (2019), as well as the Sturmian comparison theorems and transformation theorems on compact intervals by J. Elyseeva (2016 and 2018). We note that all the results are new even in the completely controllable case. © 2020 Elsevier Inc. All rights reserved. MSC: 34C10 Keywords: Sturmian comparison theorem; Linear Hamiltonian system; Proper focal point; Minimal principal solution; Antiprincipal solution; Comparative index This research was supported by the Czech Science Foundation under grant GA19-01246S. Corresponding author. E-mail addresses: sepitkap@math.muni.cz (P. Sepitka), hilscher@math.muni.cz (R. Simon Hilscher). https://doi.Org/10.1016/j.jde.2020.02.016 0022-0396/© 2020 Elsevier Inc. All rights reserved. P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2921 1. Introduction We consider the linear Hamiltonian differential systems y' = jU(t)y, tel, (H) y' = jn(t)y, tel, (H) where I c R is a fixed interval and H,TL: 1 R2"x2" are given piecewise continuous symmetric matrix-valued functions on 1 satisfying the Sturmian majorant condition H(t)>H(t) forallfeZ. (1.1) In this setting we say that system (H) is a Sturmian majorant of (H), or that system (H) is a Sturmian minorant of (H). We assume that n e N is a given dimension and J e R2"x2" is the canonical skew-symmetric matrix (see equation (2.12) below). In addition to (1.1) we assume that the minorant system (H) satisfies the Legendre condition B{t) >0 forallfeZ, (1.2) where B(t) is the lower right n x n block of H(t). Along with the basic systems (H) and (H) we will also consider a certain transformed linear Hamiltonian system y' = jU(t)y, tel, (H) which is related to (H) and (H) by a symplectic transformation (see Remark 1.4 below). We are interested in the Sturmian comparison theorems, which provide a way for the estimation of the number of focal points of conjoined bases of the majorant system (H) in terms of the number of focal points of conjoined bases of the minorant system (H), or vice versa. The novel approach of this paper resides in four aspects: (i) we consider an unbounded interval 1 and thus derive the singular Sturmian comparison theorems, (ii) we remove the controllability assumption on the involved systems, (iii) we obtain exact formulas for the numbers of focal points of conjoined bases of these two systems, and (iv) as key tools we derive new results in the transformation theory of principal and antiprincipal solutions at oo and new limit properties of the comparative index involving these solutions. Solutions of systems (H) or (H) are piecewise continuously differentiable functions on 1. We will consider 2n-vector-valued solutions denoted by small letters (typically y or y) or 2n x n-matrix-valued solutions denoted by capital letters (typically Y or Y). We will split the vector solutions into their n-vector components y = (xT,uT)T or the matrix solutions into their n x n-matrix components Y = (XT, UT)T, see notation (2.14) below. In the theory of uncontrollable linear Hamiltonian systems it is known that the conjoined bases may have the X-component singular on a nondegenerate subinterval of 1. More precisely, the result of [23, Theorem 3] or [18, Proof of Lemma 3.6(a)] shows that under the Legendre condition the kernel of X(t) is piecewise constant on 1 for any conjoined basis Y. This means that the kernel of X(t) changes finitely many times in any compact subinterval of 1 and we say that Y has a left proper focal point altoel if KerX(f^") ^ KerX(fo) with the multiplicity 2922 P. Šepitka, R. Šimon Hilscher / J. Differential Equations 269 (2020) 2920-2955 mL(t0) := def X(t0) - def X(t~), (1.3) and a right proper focal point at to el if KerX(f^) ^ KerX(fo) with the multiplicity mR(to) := def X(t0) - def X(t+). (1.4) These notions were defined in [24,36]. For brevity, the adjective "proper" will be disregarded in the subsequent terminology. The notations KerX^), def X(f^), and later rankX(f^) represent the one-sided limits at fo of the piecewise constant quantities KerX(f), def X(t) := dimKerX(f), and rankX(f). When counting the left and right focal points of conjoined bases Y of (H) in the interval X we will use the notation m^(X) and mR(X), that is, mL(X) := ^ mL(t0), mR(X) := ^ mR(t0). (1.5) toeT toeT In a similar way we will use the notation m^(X) and mR(X) for a conjoined basis Y of (H). For special conjoined bases YtQ and YtQ, called the principal solutions of (H) and (H) at to e Z and defined by the initial conditions Yt0(t0) = E, Yt0(t0) = E, E:=(0,I)T, (1.6) we will use the notation mu0(X), mRtQ(X), inu0(X), fnRtQ(l). The brackets around X will be dropped when considering an interval X with specific endpoints. The focal points are always counted including their multiplicities. Oscillation theory of linear Hamiltonian systems represents a classical topic in the qualitative theory of differential equations. Standard references include the monographs [4,9,19,22,25,26] or more recently [2,21,27]. Regarding the compact interval X = [a,b], the classical Sturmian comparison theorem for the second order Sturm-Liouville differential equations is presented in [19, Theorem XI.3.1] or [26, Theorem 11.3.2(a)]. An extension of this result to controllable linear Hamiltonian systems (H) and (H) was derived in [8, Theorem 4] by Coppel, in [3, pg. 252] by Arnold (also quoted in [27, Theorem 4.8]), and in [22, Section 7.3] by Kratz. In particular, the results in [22, Corollary 7.3.2] use the principal solutions of (H) and (H) at the endpoints a and b as the reference solutions for counting the focal points. We recall that system (H) is called completely controllable (or identically normal) on the interval X if the only solution y = (x T (•) = 0, uT)T of (H) on a nondegenerate subinterval To c T is the trivial solution y(-) = 0. Note that in this case the quantities in (1.3) and (1.4) coincide with the usual multiplicity of a focal point at fo> which is defined by (see [22, Theorem 3.1.2]) m(t0) := def X(t0) = dimKerX(i0). (1.7) Regarding an open or unbounded interval X, a singular Sturmian comparison theorem for the second order Sturm-Liouville differential equations was obtained in [1, Theorem l(i)] by Aharonov and Elias. Moreover, the following singular comparison theorem for controllable systems (H) and (H) on X = [a, oo) was derived in [12, Theorem 2] by Došlý and Kratz. The authors of [1,12] replaced the principal solution at b by the principal solution at oo in their comparison theorems. We recall, see e.g. [10], that the principal solution of (H) at oo is defined as its conjoined basis Yoo with Xoo(t) invertible on [a, oo) for some a e [a, oo) and P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2923 \im^jx^\s)B(s)XTT1(s)ds^ = 0. (1.8) a Proposition 1.1. Assume that (1.1) and (1.2) hold on X =[a, oo) and that systems (H) and (H) are nonoscillatory and completely controllable. Then the number of focal points of any conjoined basis Y of (H) in the interval [a, oo) is bounded from below by the number of focal points of the principal solution of (H) at oo in this interval. This means, in the above notation, m[a, oo) > moo[a, oo). (1-9) The first Sturmian comparison theorems for uncontrollable linear Hamiltonian systems were derived in [34, Theorems 1.2 and 1.3] by the second author for a compact interval X = [a,b] and for the left focal points. These results are translated easily via [24, Remark 4.7] to the right focal points. The comparison theorems in [22, Section 7.3] and [34] were derived by using the oscillation theorems for self-adjoint eigenvalue problems and they are formulated as estimates or inequalities (unless H(t) = H(t) on X). On the other hand, the following exact formula for expressing the numbers of left and right focal points of conjoined bases Y and Y of (H) and (H) on X = [a, b] was derived in [15, Theorem 2.2]. This result is based on using the comparative index /x(-, •) and the dual comparative index /x*(-, •) of Elyseeva [13,14], which we define in Section 2. Proposition 1.2. Assume that (I.I) and (1.2) hold on X = [a, b] and let Y and Y be any conjoined bases of(H) and (H). Let Z be a fundamental matrix of(H) satisfying Y(t) = Z(t) E on [a, b], where the matrix E is given in (1.6), and consider the function Y(t) := Z~l(t) Y(t) on [a,b]. Then the comparative index /x(F(f), Y(t)) is piecewise constant and right-continuous on [a, b] and for every to e (a, b] the multiplicities mi(to), frii(to), and rh^ito) of left focal points ofY, Y, and Y at to defined through (1.3) satisfy the equality mL(to) - mL(to) = mL(to) + fi(Y(t0), Y(t0)) - lim fi(Y(t), Y(t)). (1.10) Moreover, the numbers of left focal points of Y and Y in (a, b] are connected by mL(a, b] - inL(a, b] = mL(a, b] + fi(Y(b), Y(b)) - /x(F(a), F(a)), (1.11) where m^(a, b] is the number of left focal points in (a, b] of the auxiliary function Y. A corresponding result for the right focal points in [a, b) can be derived by an analogous method to the proof of Proposition 1.2 in [15]. Alternatively, we may use the relationship mi(a, b] + rankX(Z?) =mR[a,b) + rankX(a) between the left and right focal points of Y. Proposition 1.3. Under the assumptions of Proposition 1.2, for any conjoined bases Y and Y of (H) and (H) the dual comparative index /x*(F(f), Y(t)) is piecewise constant and left-continuous on [a,b] and for every to € [a,b) the multiplicities m#(?o), m#(?o), and m^(?o) of right focal points ofY, Y, and Y := Z~lY at to defined through (1.4) satisfy the equality 2924 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 mR(t0) - mR(t0) = mR(t0) + fi*(Y(t0), Y(t0)) - lim fi*(Y(t), Y(t)). (1.12) Moreover, the numbers of right focal points of Y and Y in [a, b) are connected by mR[a, b) - inR[a, b) = mR[a,b) + fi*(Y(a), Y(a)) - fi*(Y(b), Y(b)), (1.13) where mR[a,b) is the number of right focal points in [a, b) of the auxiliary function Y. Remark 1.4. We note that the symplectic fundamental matrix Z of (H) in Propositions 1.2 and 1.3 has the form Z = (*, Y). Moreover, it is easy to verify (see [11]) that the function Y := Z~lY is a conjoined basis of the transformed linear Hamiltonian system (H), whose coefficient matrix H(t) :=ZT(t)[U(t) -H(t)]Z(t), tel, (1.14) satisfies H(t) > 0 on X under (1.1). The question regarding the validity of the singular Sturmian comparison theorem for two nonoscillatory linear Hamiltonian systems (H) and (H) satisfying (1.1) on an unbounded interval 1 and no controllability condition is an open problem so far. In the present paper we solve this problem and provide a generalization of Propositions 1.1, 1.2, and 1.3 to this setting (Theorem 5.1). This extension is by no means straightforward. The investigation of this problem revealed the necessity to extend first the transformation theory of linear Hamiltonian systems involving the comparative index, known in [16,17], to unbounded intervals (Theorem 3.2). Along this way we also obtained new results regarding the transformation of the principal and antiprin-cipal solutions at oo (Theorems 3.6, 3.8, and 4.4), which play a fundamental role in our new singular Sturmian comparison theorems. As we recently observed in [33], when considering an unbounded interval 1 = [a, oo) it is essential to include the multiplicities of focal points at oo and to use the minimal principal solutions of (H) and (H) at oo as the reference solutions for counting the focal points. Following [28], the minimal principal solution of (H) at oo is defined as the conjoined basis Yoo of (H) with ^oo(0 having constant kernel on [a, oo) for some a e [a, oo) and ffim X^is) B(s) X7^ (s)dsj =0, rank (?) = n - doo, te[a,oo). (1.15) a Here ^ denotes the Moore-Penrose pseudoinverse (see [5-7]), the number doo is the maximal order of abnormality of (H), and the rank of Xoo(t) is minimal possible on [a, oo). One can see that condition (1.15) directly generalizes (1.8) to the uncontrollable setting. As applications of the main comparison theorem we derive additional exact formulas and estimates for the numbers of focal points of the principal solutions Ya, Ya and Yoo, Yoo (Theorem 5.5 and Corollaries 5.4 and 5.7). Finally, we note that all the results in this paper are new even for controllable linear Hamiltonian systems, in particular they are also new for the even order Sturm-Liouville differential equations. We are thus convinced that this paper represents an important contribution to P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2925 the qualitative theory of differential equations. Along the way to the above Sturmian comparison theorems we also discovered the necessity to complete some theoretical results from matrix analysis (Theorem A.2 in the appendix), which were initiated in [22]. The paper is organized as follows. In Section 2 we present the definition and main properties of the comparative index, as well as the needed theory for the known singular Sturmian separation theorems on 1 = [a, oo). In Section 3 we investigate the transformation theory and limit properties of the comparative index with a general symplectic transformation matrix R(t). In Section 4 we apply these results to systems (H) and (H) satisfying majorant condition (1.1) and to the special transformation matrix R(t) := Zoo(f), being the symplectic fundamental matrix of the minorant system (H) associated with the minimal principal solution Foo of (H) at oo. In Section 5 we present the main results of this paper - singular Sturmian comparison theorems on the unbounded interval 1 = [a, oo), while in Section 6 we present analogous results for the unbounded interval (—oo, b]. Finally, in Appendix A we derive a completion of some known results from matrix analysis related to normalized conjoined bases. 2. Main tools and auxiliary results The main results of this paper are based on the notion of a comparative index of two conjoined bases of (H) and (H). Following [13, Definition 2.1] or [14, Definition 2.1], for two real constant 2n x n matrices F and Y such that YTJY = 0, YTJY = 0, rankF = « = rankF, W:=YTJY (2.1) we define their comparative index /x(F, Y) and the dual comparative index /x*(F, F) by /x(F, Y) := rankM + indV, /x*(F, F) :=rankA4 + ind(—P), (2.2) where M. and V are the n x n matrices M := (/ - AfA) W, V:=VWTXfXV, V := I -MfM. (2.3) The matrices F and F are partitioned into the n x n blocks according to the standard notation Y = (XT,UT)T and Y = (XT,UT)T. We note that the matrix V is the orthogonal projector onto KerM. and the matrix V is symmetric, see [14, Theorem 2.1]. The quantity indP denotes the index of V, i.e., the number of its negative eigenvalues. Obviously, the relation ind(—V) = mnkV — indV holds. The needed algebraic properties of the comparative index are summarized as follows, see [14, Section 2]. Proposition 2.1. Let Y and Y be 2n x n matrices satisfying (2.1) and let E be given in (1.6). Then the comparative index and the dual comparative index defined in (2.2) satisfy lx(Y, Y) + ii* (Y, Y) = rank W - rank A + rank A, (2.4) max{/x(F, Y), /x*(F, F)} 0 forallfeZ (2.13) as well. By A > 0 we mean that the symmetric matrix A is positive semidefinite and for symmetric matrices A and B the notation A > B means that A — B > 0. Similarly to (2.12) we will split the matrix solutions of (H) and (H) into their n x n-matrix components as '«-$!)■ ™-(5S)- '«-(&,)■ We will be particularly interested in conjoined bases of (H) and (H), i.e., the solutions F satisfying YT (0 J F(0 = 0 and rank Y(t) = n at some (and hence at all) points tel. Next we recall several important results regarding the unbounded interval 1 = [a, 00) and the corresponding Sturmian separation theorems from [33]. System (H) is defined to be nonoscilla-tory at 00 if for some conjoined basis F of (H) (or for every conjoined basis F of (H) by [35, Theorem 2.2]) there are no left focal points of F in (a, 00) for some a e [a, 00). In this case the number a can be chosen so that the kernel of X(t) is constant on [a, 00). With a slight abuse in the terminology we will say in this case that the conjoined basis F itself has a constant kernel on [a, 00). Then by [23, Lemma 2] we also have X(t)X\t)B(t) = B(t) = B(t)X(t)X\t), te[a,oc). (2.15) Note that we may equivalently define the nonoscillation of F at 00 in terms of the nonexistence of the right focal points in [a, 00), since by [31, Theorem 5.1] we have the equality mL(a, ft] +rankX(/6) = mR[a, ft) + rankX(a), a, ft e [a, 00), a < ft. (2.16) In particular, if rankX(0 is constant on [a, 00), then (2.16) yields m^(a, 00) = mR[a, 00). P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2927 The maximal order of abnormality of (H) at oo is defined as the number doo:= lim d[t, oo) = max d[t,oo), 0 to a pair of normalized conjoined bases, see e.g. [22, Proposition 4.1.1]. Then every conjoined basis Y of (H) can be uniquely represented by a constant 2n x n matrix Coo satisfying Y(t) = Z00(t)C00, tel, c^ = [~w(f^Y))- (2'23) Observe that the results below include the multiplicities of focal points at oo, according to the notation introduced in (1.5). Proposition 2.4 (Singular Sturmian separation theorem). Assume that (2.13) holds on the interval I =[a, oo) and system (H) is nonoscillatory at oo. Then for any conjoined bases Y and Y of (H) we have the equalities mL(a, oo] - mL(a, oo] = /x(Coo, Coo) - /x(F(a), Y(a)), (2.24) = /x(F(a), Foo(a)) - /x(F(a), Y^a)), (2.25) mR[a, oo) — m,R\a, oo) = /x*(F(a), Y(a)) — /x*(Coo, Coo), (2.26) = fi*(Y(a), Foo(a)) - /x*(F(a), Y^(a)), (2.27) where Coo and Coo are the constant matrices in (2.23) corresponding to Y and Y. Proof. Formulas (2.24) and (2.26) were proven in [33, Theorem 5.1]. Next we use (2.24) for the conjoined bases Y and Foo (i-e., with the representation matrices Coo and E), and then we use (2.24) again for the conjoined bases Y and F>o (i-e., with the representation matrices Coo and E). Subtracting the resulting equalities yields formula (2.25). Similarly, applying (2.26) once to Y and Foo and once to F and Fxd and subtracting the outcome leads to formula (2.27). □ P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2929 Remark 2.5. We note that in [33] we used the alternative symplectic fundamental matrix Ooo = (Too, *) and the representation Y(t) = <&oo(t) Doo on X, which yields that Coo = —JD-But since by property (2.7) with M = —I = M we have [i(JDoo, JDoo) = p(Coo, Coo) and Ijl*(JDoo, JDoo) = P*(Coo, Coo), all the formulas in [33] involving the comparative index (or the dual comparative index) with JDoo can be replaced by the same formulas involving the matrices Coo- For the special choice of Y := Ya (being the principal solution at a) or 7 != Yoo (being the minimal principal solution at oo) the results in Proposition 2.4 yield the following, see [33, Corollary 5.3 and Theorems 5.5 and 5.7]. Proposition 2.6 (Singular Sturmian separation theorem). Assume that (2.13) holds on the interval X = [a, oo) and system (H) is nonoscillatory at oo. Then for any conjoined basis Y of(R) we have the equalities mLa(a, oo] + /x(Coo, C^) = mL(a, oo] = mLoo(a, oo] - fi(Y(a), Foo(«)), (2.28) mRoo[a, oo) + ij*(Y(o), Foo(a)) = mR[a, oo) = mRa[a, oo) - /x*(Coo, C^), (2.29) where Coo and are the constant matrices in (2.23) corresponding to Y and Ya. Moreover, m^a (a, oo] < »i£ (a, oo] < m^oo («, oo], (2.30) wiRooia, oo) < mR[a, oo) < mRa[a, oo), (2.31) and the above lower and upper bounds are related by the equalities wiLooia, oo] = m^a(a, oo] + rankX00(a), mRa[a, oo) = m^oofa, oo) +rankX00(a), (2.32) mLa(a, oo] = mRoo[a, oo), mLoo(a, oo] = mRa[a, oo). (2.33) Finally, for completeness we present a formula relating the numbers of left and right focal points of one conjoined basis of (H) in an unbounded interval, see [33, Theorems 5.4]. This formula also involves the minimal principal solution Yoo at oo, as well as the principal solution Ya at the left endpoint a, since X(a) = — W(Ya, Y). Proposition 2.7. Assume that (2.13) holds with X = [a, oo) and system (H) is nonoscillatory at oo. Then for any conjoined basis Y of(R) we have the equality mi(a, oo] +rankWr(F00, Y) =mR[a, oo) +rankX(a). (2.34) Formula (2.34) will be used in the proof of the singular comparison theorem for the right focal points, knowing the result for the left focal points. The results in Propositions 2.4-2.7 highlight the symmetric role of the (minimal) principal solutions Ya and Yoo in the Sturmian separation theorems for one system (H) on the interval X = [a, oo). Similar symmetry holds also for other types of unbounded intervals X = (—oo, b] or X = (—oo, oo) and the (minimal) principal solutions at their endpoints, see [33, Remark 8.1]. 2930 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 3. Transformation and limit results for comparative index In this section we consider a nonoscillatory linear Hamiltonian system (H) on 1 = [a, oo) and a given piecewise continuously differentiable 2n x 2n symplectic transformation matrix R(t) on the interval 1. Analogously to Remark 1.4, the transformation y:=R-Ht)y (3.1) transforms system (H) into linear Hamiltonian system (H) with the coefficient matrix U(t) ;=RT(t)[U(t) -UR (t)]R(t), UR(t):=JR'(t)JRT(t)J, tel. (3.2) We will assume in this section that the transformed system (H) is nonoscillatory at oo and satisfies the Legendre condition B(t)>0 forallfeZ, B(t) := ET J-L(t) E, (3.3) with the matrix E given in (1.6). Our aim is to establish for a given conjoined basis Y of (H) limit results at oo for the comparative indices fi(Y(t),R(t)E) and fi*(Y(t), R(t)E), (3.4) and to find conditions under which the minimal principal solution of (H) at oo transforms to the minimal principal solution of (H) at oo. Since the matrix R(t) is piecewise continuously differentiable and symplectic on X, the matrix function R(t)E appearing in (3.4) is a conjoined basis of the linear Hamiltonian system y' = JUR(t)y, tel, (HR) with the symmetric coefficient matrix HR(t) defined in (3.2). When R(t) is a fundamental matrix of the original system (H), then of course HR(t) = H(t) and R(t)E is a conjoined basis of (H). In this case the limit results at oo for (3.4) are known in [33, Theorem 6.1]. Proposition 3.1. Assume that (2.13) holds with I = [a, oo) and system (H) is nonoscillatory at oo. Let Y be a conjoined basis of (H) and let R(t) be a symplectic fundamental matrix of (H). Then for Y*(t) := R(t)E we have lim fi(Y(t), Y*(t)) = /x(Coo, O - mL(oo) + m* (oo), (3.5) t —>oo lim fi*(Y(t), Y*(t)) = /x*(Coo, O, (3.6) where Coo and are the constant matrices in (2.23) corresponding to Y and Y*, and where m*L(co) is the multiplicity of a focal point at oo of the conjoined basis Y*. P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2931 In the main result of this section (Theorem 3.2 below) we generalize Proposition 3.1 to an arbitrary piecewise continuously differentiable symplectic matrix R(t). For convenience and in accordance with [17, Eq. (1.3)] we split the matrix R(t) into the n x n blocks as We recall the notation Foo and Fxi for the minimal principal solutions of (H) and (H) at oo. Theorem 3.2. Assume that (2.13) holds with X = [a, oo), system (H) is nonoscillatory at oo, and Y is a conjoined basis of (H). Let R(t) be a piecewise continuously differentiable symplectic matrix on X with the partition in (3.7) and assume that system (H) with (3.2) satisfies (3.3) and it is nonoscillatory at oo. Then the limits lim ix(Y(t),R(t)E) and lim fi*(Y(t), R(t)E) (3.8) t^oo t^oo both exist if and only if rankM(f) is eventually constant, (3.9) i.e., the limit of'rank M(t) exists for t -> oo. In this case the limits lim ^(R-^OY^iO^-^^E) and lim fi*(R(t) Y^t), R(t)E) (3.10) also exist and we have the equalities lim ft(Y(t), R(t)E) = /x(Coo, Coo,/?) - mL(oc) + mL(oo) t^oo + lim ix*(R-\t) Foo(0, R-\t)E), (3.11) lim fi*(Y(t), R(t)E) = /x*(Coo, Coo,/?) + lim fi*(R(t) Y^t), R(t)E). (3.12) Here Coo Coo,/? «^ the constant matrices in (2.23) corresponding to Y and R Foo. and m^(oo) is the multiplicity of a focal point at oo of the conjoined basis Y := R~l Y of the transformed system (H). The remaining part of this section (except of the last result presented in Corollary 3.11) will be devoted to developing the necessary tools for the proof of Theorem 3.2. First we present several comments and needed results, which turn to be important on their own independently of their future applications in Sections 4 and 5. Remark 3.3. (i) Condition (3.9) in Theorem 3.2 is independent of Y. Therefore, when the limits in (3.8) exist for one conjoined basis Y of (H), then these limits exist for every conjoined basis Y of (H). Moreover, condition (3.9) guarantees through [16, Theorem 2.2] that the oscillation properties of systems (H) and (H) are preserved, that is, under (2.13) and (3.3) systems (H) and (H) oscillate or do not oscillate at oo simultaneously. 2932 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 (ii) In [17, Theorem 2.5 and Remark 2.6(h)], the existence of the limits in (3.8) at a finite point to e T is proven under the sufficient condition that the rank of M(t) is constant in a left and right deleted neighborhood of to. The statement in Theorem 3.2 can be regarded as an analogue of those results for to = oo, namely formula (3.11) corresponds to [17, Eq. (2.11)] and formula (3.12) corresponds to the second part of [17, Eq. (2.17)]. At the same time, the proof of Theorem 3.2 will show that the stated condition on the constant rank of M(t) in [17, Theorem 2.5] is not only sufficient, but also necessary. (hi) If in addition the system (Hr) satisfies the Legendre condition 5r(0 := ET tlr(t) E > 0 for all t e T = [a, oo), then condition (3.9) is equivalent with Ker M(t) being eventually constant, and hence (3.9) is equivalent with the nonoscillation of system (Hr) at oo. For a conjoined basis 7 of (H) and a point t el we define the quantities q(Y, t) := fi(Y(t), R(t)E) - /x(Coo, C^R) + mL(oo) - mL(oo), (3.13) q*(Y, t) := fi*(Y(t), R(t)E) - /x*(Coo, C^R), (3.14) where mi(oo) and in l(oo) are the multiplicities of the focal point at oo of Y and 7 := R~lY, and where Coo and Coo,# are the constant matrices in (2.23) corresponding to Y and R Too - By using property (2.4) of the comparative index and formula (2.21) for the multiplicity of a focal point at oo we then obtain lim {rankM(f) -q(Y,t) -q*(Y,t)} = rank W^Txd, # 7>o). (3.15) This formula represents an analogue of [17, Eq. (2.16)] for the case of to = oo. The quantities q(Y, t) and q*(Y, t) allow to interpret Theorem 3.2 in a simpler form and to shorten its proof. Remark 3.4. With the aid of the quantities q(Y, t) and q*(Y, t) defined in (3.13) and (3.14) we may reformulate the result in Theorem 3.2 as follows. The limits lim (?(F,0 and Mm q*(Y,t) (3.16) both exist if and only if the rank of M(t) is eventually constant, and in this case lim q(Y, t) = lim /x*(tf_1(0 1^,(0, R~\t)E), (3.17) lim q*(Y, t) = lim fi*(R(t) Y^t), R(t)E). (3.18) Formulas (3.17) and (3.18) confirm that the values of the limits in (3.16) do not depend on the choice of the conjoined basis 7 of (H). Remark 3.5. When R(t) is a symplectic fundamental matrix of system (H), then the matrix hr(t) = H(t) in (3.2), and hence H(t) = 0. In this case all solutions of system (H) are constant on X, in particular Foo(0 = E and mi(oo) = 0 (compare with [28, Example 8.2] and [33, Remark 3.2(h)]). Then we have lim ii*(R-1(t)Y&0(t),R-1(t)E)=m*(oo), lim fi*(R(t) Y^t), R(t)E) = 0, (3.19) P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2933 where m*L(oo) is the multiplicity of a focal point at oo of the conjoined basis F* := RE of (H). This can be seen as follows. The second limit in (3.19) holds trivially. For the first limit we apply property (2.11) (with V := R~x(t), Z\ := Zoo(t), and Z2 := I) to get fi*(R-1(t)Yx(t),R-1(t)E)=Tai±M(t)-fi*(Y(X(t),R(t)E), tel. (3.20) Then by taking the limit for t —>- oo and using equality (3.6) in Proposition 3.1 (with F := Foo and Coo : = £), we obtain lim ix*{R~l(t) Foo(f), R~l(t)E) = lim rankM(f) - fi*(E, C*) (2.6)^2.23) ^ mnkM(f) _ rank w(y y+) (2.21) * (oq)_ This shows that when R(t) is a symplectic fundamental matrix of system (H), then Theorem 3.2 reduces exactly to Proposition 3.1. Note also that in this case (3.17) and (3.18) have the form lim q(Y,t) = m*r(oo), lim q*(Y, t) = 0. (3.21) In the proof of Theorem 3.2 we will utilize a similar representation of conjoined bases of the transformed system (H) as in (2.22) and (2.23). That is, we consider the symplectic fundamental matrix Zoo of (H) associated with the minimal principal solution Foo of (H) at oo through the equality Foo(0 = Zoo(0 E on 1. Then every conjoined basis Y of (H) can be uniquely represented by a constant 2n x n matrix Coo satisfying Y{t) = ZO0{t)CO0, tel, (I, 0)Coo = -W(Foo,F). (3.22) In the next statement we consider the transformation of maximal antiprincipal solutions of (H) at oo under (3.1). In particular, we describe when a maximal antiprincipal solution of (H) at oo is transformed into a maximal antiprincipal solution of (H) at oo. Theorem 3.6. Assume that (2.13) holds with I =[a, oo), system (H) is nonoscillatory at oo, and Y is a conjoined basis of (H). Let R(t) be a piecewise continuously differentiable symplectic matrix on I and assume that system (H) with (3.2) satisfies (3.3) and it is nonoscillatory at oo. Moreover, let Coo and Coo,« be the constant matrices in (2.23) corresponding to Y and R Foo, and let F^r and D^r be the matrices in (A.2) and (A.3) in Appendix A corresponding to the constant 2n x n matrix Coo,«- Then the following statements are equivalent. (i) The conjoined bases Y and Y := R~lY are maximal antiprincipal solutions at oo of systems (H) and (H), respectively. (ii) The matrix Coo has the form Coo = (-JCocr F^ + Coo,« D) K, (3.23) where D and K are constant n x n matrices such that K is invertible, D is symmetric, Im{W(Y00, R Foo) (D - D^R) [ W(Y0Q, R Y^)]7} = Im W(Y0Q, R Y^). (3.24) 2934 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 In this case K = WXFxd, Y), the matrix W(Y(t), R(t)E) is invertiblefor large t el, and ind {W(Foo, R Foo)(X> - D^r) [ W(Yoo, R Yoc)f} = ind {W(Foo, F) [ W(Foo, F)]"1 W(Foo, R Proof. The proof is based on the characterization of a maximal antiprincipal solution at oo in Proposition 2.3. More precisely, the conjoined bases Y and Y := R~l Y are maximal antiprincipal solutions at oo of systems (H) and (H), respectively, if and only if the Wronskians WXFxd, Y) and W(Foo> Y) are invertible. Since the matrices Z^1 (?) and R(t) are symplectic on 1, we obtain (suppressing the argument t el) W(C0o^r,C0o) = CooRJC00 = Y^R Z^ JZ00Y = Y00R JRR Y = Y^JR~lY = ^(Foo, R~lY) = ^(Foo, F). (3.26) Moreover, since by (2.23) we have (/, 0) Coo = - WiY^, Y) and (/, 0) Coo,« = - WQoo, # ?co), the upper « x « block of Coo is invertible. Therefore, Coo,# and Coo [^(Foo, F)]_1 are constant matrices satisfying the properties required in Theorem A.2 in the appendix (with F := Coo,# and F := Coo [W(Yoo, F)]_1). Hence, Theorem A.2 yields that statement (i) is equivalent with the fact that Coo has the form in (3.23) with K = W(Yoo, Y) and D satisfying (3.24). Finally, in this case (3.25) holds by (A.5) in Theorem A.2, and the matrix W(Y(t), R(t)E) = WiR'Ht) Y(t), E) = W(Y(t), E) = XT(t) (3.27) is invertible for large t el, since Y is a maximal conjoined basis of (H). □ Remark 3.7. Based on Remark A.3(ii) in the appendix we may conclude that for every transformation matrix R(t) in Theorem 3.6 there always exists a maximal antiprincipal solution F of (H) at oo, which is transformed to the maximal antiprincipal solution F := R~lY of (H) at oo. For the associated representation matrices Coo and Cqqr of F and R Yqq in Theorem 3.6 we then have, by the definition in (2.2), /x(Coo, Coo,/?) = ind{-W(Foo, F) [ W(Foo, F)]"1 WiY^, RY^)}, (3.28) /x*(Coo, Coo,*) = ind {W(Foo, F) [ W(Foo, Y)]-1 W(Y^, RY^)}. (3.29) In these calculations we used that W(Coo, C^r) = -[ WXFxd, Y)]T obtained from (3.26). The statement in Theorem 3.6 yields the following result regarding the transformation of maximal antiprincipal solutions of (H) and (H) at oo, as well as the transformation of minimal principal solutions of (H) and (H) at oo. Theorem 3.8. Assume that (2.13) holds with 1 =[a, oo), system (H) is nonoscillatory at oo, and Y is a conjoined basis of (H). Let R(t) be a piecewise continuously differentiable symplectic matrix on 1 and assume that system (H) with (3.2) satisfies (3.3) and it is nonoscillatory at oo. Then the following statements are equivalent. (3.25) P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2935 (i) Every maximal antiprincipal solution of (H) at oo is transformed into a maximal antiprin-cipal solution of(H) at oo. (ii) Every maximal antiprincipal solution of (H) at oo is a transformation of some maximal antiprincipal solution of (H) at oo. (iii) The minimal principal solution of (H) at oo is transformed into the minimal principal solution of (H) at oo. (iv) The constant Wronskian matrices W(R~1Y00, ?oo) ond W(loo, RYqq) satisfy WiR^Yoo, Foo) = 0 = WiY^RYoo). Proof. Clearly, conditions (iii) and (iv) are equivalent. Next we show that (i) is equivalent with condition W(Yoo, R Foo) = 0 in (iv). Assume that (i) holds. Then by Theorem 3.6 condition (3.24) is satisfied for every symmetric matrix D. In particular, for D := Dqor in (3.24) we obtain that W{Yoo, RYqo) = 0. Conversely, if (iv) holds, then condition (3.24) is trivially satisfied for every symmetric matrix D, so that condition (i) follows from Theorem 3.6. Finally, in a similar way we prove that (ii) is equivalent with condition W(R~1Y00, Foo) = 0 in (iv), since the system (H) is transformed into system (H) by the symplectic transformation matrix 7\_1(f). The proof is complete. □ In the next result we examine the limit properties of the comparative index and the dual comparative index in (3.8), when F is a maximal antiprincipal solution of (H) at oo satisfying the properties in Theorem 3.6 and Remark 3.7. Theorem 3.9. Assume that (2.13) holds with I =[a, oo), system (H) is nonoscillatory at oo. Let R(t) be a piecewise continuously differentiable symplectic matrix on I with partition (3.7) and assume that system (H) with (3.2) satisfies (3.3) and it is nonoscillatory at oo. Moreover, let Y be a maximal antiprincipal solution of(H) at oo such that Y := R~lY is a maximal antiprincipal solution of(H) at oo. Then the two limits in (3.8) exist if and only if the limit of the rank of M(t) for t -> oo exists, i.e., condition (3.9) holds. In this case we have the formulas lim fi(Y(t),R(t)E) = lim wd\[W(Y(t), R(t)E)f X~\t)M(t)\, (3.30) lim ix*(Y(t),R(t)E) = lim ind \-[W(Y(t), R(t)E)f X'^t) M(t)\. (3.31) Proof. Let F and Y be as in the theorem. Choose a e [a, oo) such that X(t) and X(t) are invertible on [a, oo). Applying the definition of the (dual) comparative index in (2.2) with (2.3) we obtain that fi(Y(t),R(t)E) = indV(t), fi*(Y(t), R(t)E) =ind[-V(t)], V(t) := [W(Y(t), R(t)E)]TX~Ht) M(t), Since by Theorem 3.6 and (3.27) we know that the Wronskian W(Y(t), R(t)E) is invertible on [a, oo), it follows that te[a,oc). (3.32) mdV(t)+md[-V(t)] = mnkV(t) = rankM(f), te[a,oc). (3.33) 2936 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 Due to the continuity of V{t) and its eigenvalues, the equality in (3.33) implies that the quantities indP(f) and ind [—V(t)] are eventually constant (i.e., their limits for t —>- oo exist) if and only if condition (3.9) holds. Consequently, by (3.32) the two limits in (3.8) exist if and only if condition (3.9) is satisfied, and in this case (3.30) and (3.31) hold. □ In the following auxiliary result we study the behavior of the (dual) comparative index in (3.4) for two conjoined bases Y\ and Y2 of (H) for large tel. We derive an analogue of [16, Lemma 3.2] for the case of to = 00. Lemma 3.10. Assume that (2.13) holds with I =[a, 00), system (H) is nonoscillatory at 00. Let R(t) be a piecewise continuously differentiable symplectic matrix on X and assume that system (H) with (3.2) satisfies (3.3) and it is nonoscillatory at 00. Let Y\ and Y2 be conjoined bases of (H) and let Cioo, C200, and Coo,# be the constant representation matrices in (2.23) associated with Y\, Y2, and R Foo- Then there exists a e [a, 00) such that for all t e [a, 00) /x(Fi(f), R(t)E) - (m(Cioo,Coo,r) +m1L(oo) - m1L(oo) = fi(Y2(t), R{t)E) - /x(C20o, Coo,/?) +m2L(oo) - m2L(oo), (3.34) /x*(Yi(0, R(t)E) - /x*(Cl0O, Coo,/?) = ti*(Y2(t), R(t)E) - /x*(C2oo, C^R), (3.35) where m\i(oo), w/2l(oo) and m\i(oo), w/2l(oo) are the multiplicities of the focal point at 00 of the conjoined bases Y\, Y2 and Y\ := R~lY\, Y2 '■= R~lY2- Equivalently, q(Yut)=q(Y2,t) and q*(Y1,t)=q*(Y2,t) for all t e [a, 00). (3.36) Proof. We proceed in a similar way to the proof of [16, Lemma 3.2]. Let Y\ and Y2 be given conjoined bases of (H). Denote by Y\ := R~lY\ and Y2 := R~lY2 the corresponding conjoined bases of system (H), and consider the symplectic fundamental matrices Z\ and Z2 of (H) satisfying Z\E = Y\ and Z2E = Y2. Fix now t e T. By formulas (2.8) and (2.10) in Proposition 2.2 (with V := R(t), Z\ := Z\(t), Z2 := Z2(0) we obtain the identities KYiit), R(t)E) - ii{Y2{t), R(t)E) = ii{Yx{t), Y2(t)) - /*(fi(0, Y2(t)), (3.37) fiHYtit), R(t)E) - fi*(Y2(t), R(t)E) = fi*(Y!(t), Y2(t)) - ^(Y^t), Y2(t)). (3.38) By Proposition 3.1 we know that the limits of the comparative indices on the right-hand side of (3.37) and (3.38) exist with lim (71(0,72(0) t^oo = P(C\oo, C200) — m\L{oo) +m2L(oo), (3.39) lim /x(Fi(0,?2(0) t^oo = P(C\oo, C200) — miL{oo) +m2L(°o), (3.40) lim /x*(Yi(0,F2(0) ^00 = At*(Cioo, C200), (3.41) lim /x*(Fi(0,F2(0) = At*(Cioo, C200), (3.42) P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2937 where Cioo and C2oo are the constant matrices in (3.22) associated with Y\ and Y2. This implies that there exists a el such that the (dual) comparative indices in (3.39)-(3.42) are constant on [a, 00), and hence on this interval we have li(Yi(t), R(t)E) - /x(F2(0, R(t)E) = /x(Cl0O, C20o) - /x(Cl0O, C2oo) - m\L(oo) + m2L(oo) + m\L(oo) — m2L(oo), (3.43) /x*(Fi(0, R(t)E) - fi*(Y2(t), R(t)E) = fi*(Cl00, C2oo) - fi*(Cl00, C2oo). (3.44) Fix now t e [a, 00). Applying formulas (2.8) and (2.10) in Proposition 2.2 again (this time with V := Z^(t) R(t) Zoo(f), Z\ := Z^(t) Zx(t), Z2 := Z"1^) Z2(0) and using the equalities Z^Yi = Cioo, Z"*F2 = C2oo, Z"1?! = Cioo, Z^Y2 = C2oo, and Z^Rfx = C^r we get //.(Cioo, C2oo) — //(Cioo, C2oo) = //(Cioo, Coo,«) — /z(C2oo, Coo,«), (3.45) //*(Cioo, C2oo) — //*(Cioo, C2oo) = //*(Cioo, Coo,«) — //*(C2oo, Coo,«)- (3.46) Upon inserting (3.45) into (3.43) we obtain the equality in (3.34) for all t e [a, 00), while using (3.46) in (3.44) yields the equality in (3.35) for all t e [a, 00). Finally, the proof of (3.36) follows from the definition of q(Y, t) and q*(Y, t) in (3.13) and (3.14). □ We are now ready to present the proof of Theorem 3.2. Proof of Theorem 3.2. Let Y be a conjoined basis of (H) and let R be a symplectic matrix satisfying the assumptions of the theorem. From Remark 3.7 we know that there exists a maximal antiprincipal solution Y of (H) at 00 such that R~l Y is a maximal antiprincipal solution of system (H) at 00. By equations (3.34) and (3.35) in Lemma 3.10 (with Fi := Y and F2 := Y), the limits in (3.8) exist if and only if the limits lim /x(Y(t),R(t)E) and lim fi*(Y(t), R(t)E) both exist. In turn, this is equivalent by Theorem 3.9 (with Y := Y) to rankM(f) being eventually constant. This proves the first part of Theorem 3.2. Next, under condition (3.9), we will prove (3.11) and (3.12) by deriving their equivalent form (3.17) and (3.18). We know by (3.36) in Lemma 3.10 that the limits of the quantities q(Y\, t) and q*(Y\, t) foxt^oo exist and their values do not depend on the choice of the conjoined basis Fi. For the proof of (3.18) we take two choices of Fi := F and Fi := R Yqq. In the latter case Cioo = Coo,# and hence /x*(Cioo, Coo,#) = 0. Therefore, we calculate lim q*(Y, t) = lim q*(RY00,t) (3=4) lim fi*(R(t) Foo(f), R(t)E). This proves formula (3.18), and hence also formula (3.12). For the proof of formula (3.17) we take two choices of Fi := F and Fi := Fxd. In the latter case Cioo = E and from properties (2.6) and (2.23) we obtain /x(Cioo, Coo,r) =rankWr(Fo0, T^Foo) and 2938 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 lim q*(Y00,t) (=4) lim /x*(Foo(0, R(t)E) - rank W(Fqo, R ?«,). (3.47) Therefore, we conclude that lim q(Y, t)= lim ^(Foo,?) (3=5) lim rankM(f) - lim q*(Y0Q, t) - rank WXFxd, R ?«,) (3=7) lim rankM(f) - lim fi*(Yoo(t), R(t)E) {~=> lim ^(R-itfYooit), R-H^E). This proves formula (3.17), and hence also formula (3.11). The proof is complete. □ In the final result of this section we apply Theorem 3.2 to the maximal antiprincipal solution of (H) at oo used in the statement of Theorem 3.9. In this case we obtain the following explicit formulas for the limits in (3.10). They will be needed in the proof of the main transformation result for the minimal principal solutions at oo in Theorem 4.4. Corollary 3.11. Assume that (2.13) holds with X = [a, oo), system (H) is nonoscillatory at oo. Let R(t) be a piecewise continuously differentiable symplectic matrix on I with partition (3.7) such that condition (3.9) holds. In addition, assume that system (H) with (3.2) satisfies (3.3) and it is nonoscillatory at oo. Moreover, let Y be a maximal antiprincipal solution of(H) at oo such that Y := R~lY is a maximal antiprincipal solution of(H) at oo. Then the limits in (3.10) exist with lim/x*(^(0 Foo(0, tf"1^) t^oo = lim indf[W(Y(t), R(t)E)f X~\t) M(t)\ -ind{ - WiYoo, Y) [ W(Foo, F)]"1 W(Foo, RYoo)} (3.48) lim/x*(tf(0Foo(0, #(0£) t^oo = lim md\-[W(Y(t),R(t)E)]TX~1(t)M(t)\ - ind {W(?oo, Y) [ W(Foo, Y)Vl Woo, R Yoo)}■ (3.49) Proof. Formula (3.48) follows from equation (3.11) by using (3.30), (3.28), mi(oo) = 0, and ml(oo) = 0. Formula (3.49) follows from the combination of (3.12) with (3.31) and (3.29). □ 4. Transformation results under majorant condition In this section we apply the results from Section 3 to linear Hamiltonian systems (H) and (H) satisfying the majorant condition (1.1). The transformed system (H) is now obtained by the special choice of the transformation matrix R(t) := Z(t), being the symplectic fundamental matrix of the minorant system (H) (as in Propositions 1.2 and 1.3 and in Remark 1.4). In particular, in some results it will be convenient to take the transformation matrix R(t) := Zoo(0> which is associated with the minimal principal solution Fxj of (H) at oo by Fxj = Zoo E. P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2939 First we recall a comparison result for nonoscillatory systems (H) and (H) at oo under the majorant condition (1.1), as well as the invariance of the nonoscillation at oo for system (H) and the transformed system (H). The latter result is based on the generalized reciprocity principle in [16, Theorem 2.2], see also Remark 3.3(i). Proposition 4.1. Assume that (1.1) and (1.2) hold on X = [a, oo) and that system (H) is nonoscillatory at oo. Then (i) system (H) is nonoscillatory at oo, and (ii) for every symplectic fundamental matrix Z of (H) the transformed system (H) with the coefficient matrix 'Hit) given in (1.14) is nonoscillatory at oo. Proof. Recall that assumption (1.2) implies under (1.1) the Legendre condition (2.13), and that (1.1) itself implies the Legendre condition (3.3) for the transformed system (H). The nonoscillation of system (H) at oo follows from the nonoscillation of (H) at oo by [35, Theorem 2.6]. Let Z be a symplectic fundamental matrix of system (H). Then the right upper block M(t) := (7, 0) Z(r) E of the transformation matrix Z(r) has eventually constant kernel (as system (H) is nonoscillatory at oo), and hence M(t) has also eventually constant rank. Then by Remark 3.3(i) it follows that systems (H) and (H) are simultaneously oscillatory or nonoscillatory at oo. But since the nonoscillation of (H) at oo is a standing assumption of the proposition, it follows that the transformed system (H) is nonoscillatory at oo as well. □ Next we present the following auxiliary result. Lemma 4.2. Assume that (1.1) and (1.2) hold on I =[a, oo) and that system (H) is nonoscillatory at oo. Let Z be a symplectic fundamental matrix of (H). Let Y be a maximal antiprincipal solution ofQA) at oo such that Y := Z~lY is a maximal antiprincipal solution at oo of the transformed system (H) from Remark 1.4. Assume that a e [a, oo) is such that X(t) is invertible on [a, oo), the conjoined basis Y := ZE of(H) has constant kernel on [a, oo), and the Wronskian matrix W(Y(t), fit)) is invertible on [a, oo). If we set for t e [a, oo) t V(t):=[W(Y(t),Y(t))]TX-1(t)X(t), S(t):= j X\s) Bis) XfT is) ds, (4.1) a then the matrix V(t) is nondecreasing on [a, oo) and the matrix it) + Sit) is nonincreasing on [a, oo). Proof. The assumptions of the lemma imply that the Legendre conditions (2.13) and (3.3) hold and that systems (H) and (H) are nonoscillatory at oo, by Proposition 4.1. Then there exists a e [a, oo) such that X(t) is invertible on [a, oo) and the conjoined basis Y has constant kernel on [a, oo). Moreover, by Theorem 3.6 (with Rit) := Z(t)) the point a can be chosen so that the Wronskian matrix W(Y(t), fit)) is invertible on [a, oo). Then we also have ImVit) = ImXTit) = [KerXit)]1- is constant on [a, oo), (4.2) X(t)Xf(t)B(t) = B(t) = B(t)X(t)Xf(t), re [a, oo), (4.3) 2940 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 where (4.3) follows from (2.15) for system (H). Observe that indV(t) = fi(Y(t), fit)) by (3.32). Equation (4.2) yields that ImV(t) = ImP is constant on [a, oo), where P := if(r) X(t) = V(t) V\t) = V\t) Tit), t e [a, oo), is the orthogonal projector onto the constant subspace ImXT(t) = [KerX(r)]-1- on [a, oo). Thus, by [28, Theorem 4.2(iii)], we have ImS(r) c ImP on [a, oo), which is equivalent to V\t)V{t)S{t) = PS{t) = S{t) = S{t)P = S{t)V{t)V\t), re [a, oo). (4.4) According to [22, Proposition 1.1.3] and (4.3), the derivative of V(t) satisfies V'(t) = V(t) X\t)B(t)XfT(t)V(t) + V2T(t) [Hit)-Hit)] V2(t) (4=1} V(t) S\t) V(t) + V2T(t) [H{t) - H(t)] V2(t) (4.5) for f e [a, oo), where V2(t) := (I, Q(t))TX(t) and Q(t) := U(t) X~l(t). Therefore, V(t) > 0 on [a, oo) and hence the matrix V(t) is nondecreasing on the interval [a, oo). From (4.2) and [20, Theorem 20.8.2] it follows that the matrix V^(t) is differentiable on [a, oo) with [VHt)Y = -V\t)V'(t)V\t) (4-5L(4l4) -S'(t) - V\t) V2T(t) [H{t) - H(t)] V2(t)V\t) on [a, oo), see also [28, Remark 2.3]. This yields by assumption (1.1) that [V\t) + 5(0]' = ~V\t) V2T(t) [%{t) - Hit)] V2it) V\t) < 0, re [a, oo), so that the matrix V^it) + Sit) is nonincreasing on [a, oo). The proof is complete. □ For the special choice of Y := Too in Lemma 4.2 we obtain the following important property of the matrix V(t) in (4.1). Proposition 4.3. Assume that (1.1) and (1.2) hold on I =[a, oo) and that system (H) is nonoscil-latory at oo. Let be the symplectic fundamental matrix of (H) such that Y^ = Z^E. Let Y be a maximal antiprincipal solution of (H) at oo such that Y := Z^1 Y is a maximal antiprincipal solution at oo of the transformed system (H)from Remark 1.4. Then there exists a e [a, oo) such that the matrix 7?oo(0:=[^(J'(0,J>oo(0)]r^"1(0ioo(0, re[o<,oo), (4.6) satisfies Poo(0<0, re [a, oo), lim Poo(r) = 0, (4.7) t^oo lim indVooit) = lim rankXoo^) = n — doo, (4.8) where doo is the maximal order of abnormality of system (H) according to (2.17). P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2941 Proof. Let a g [a, oo) be a point satisfying the assumptions of Lemma 4.2 (with Y := Too). In particular, the Wronskian matrix W(Y(t), Yoo(0) is invertible on [a, oo). For f g [a, oo), let ^oo(0 be the matrix defined in (4.1), which corresponds to Yoq. We denote by Xj(t), Vj(t), and Kj(t) for j g {1,..., n) and t e [a, oo) the eigenvalues of the symmetric and continuous matrices ^oo(0> ~~ Soo(t), and ^(f) + Soo(t), respectively, which are ordered as Ai(f) < ••• oo(0 has constant rank r := rankPoo^) = rankXoo^) = n — doo, t e[a, 00), (4.9) and that Voo (t) is nondecreasing on this interval. Therefore, the matrix V^o (t) is nonincreasing on [a, 00). Then its eigenvalues Xj(t) are nonincreasing on [a, 00) and they do not change their sign in this interval. Next, S'^it) > 0 on [a, 00) under (1.2), so that the matrix — Soo(0 < 0 as well as its eigenvalues Vj(t) < 0 are nonincreasing on [a, 00). Moreover, by [28, Theorem 4.2(iii) and Remark 5.3] the set Im[—Soo(0] is eventually constant with - (4 9) rank[—5,30(0] = w — <^oo = r on [ft, 00) for some ft g [a, 00). Then the eigenvalues Vj (t) satisfy v\(t) < • • • < vr(t) < 0, vr+i(t) = ■■■ = vn(t) =0, te[ft,oo), \ (4.10) lim Vj(t) = — 00, j g {1,..., r}, where the last property follows from the fact that Too is a principal solution of (H) at 00, i.e., from liim^oo S^o (t) = 0. Next, by Lemma 4.2 the matrix V\t) + Soo(t) as well as its eigenvalues Kj(t) are nonincreasing on [a, 00). Therefore, Kj(t) < Kj(a) for all t e [a, 00) and j g {1,...,«}. By applying the result about the eigenvalues of a difference of two symmetric matrices in [22, Proposition 3.2.2] (with Q\ := V^o{t) and Q2 := — Soo(0) we obtain that Xj(t) — Vj(t) < Kn(t) for every t e [ft, 00) and j e {1,..., n). Hence, we get the inequalities Xj(t) °°)> so that Hindoo indT'ooCO = r- Taking (4.9) into account then completes the proof of (4.8). □ The following result shows that under natural assumptions the minimal principal solution of (H) at 00 is transformed into the minimal principal solution Foo of (H) at 00 and vice versa. We again utilize the transformation matrix R(t) := Zoo(0 associated with the minimal principal solution Foo at 00 of the minorant system (H). Theorem 4.4. Assume that (1.1) and (1.2) hold on I =[a, 00) and that system (H) is nonoscil-latory at 00. Let Zoo he the symplectic fundamental matrix of (H) such that Foo = Zoo E. Then the conjoined basis Z^1 Foo is the minimal principal solution of (H) at 00 and the conjoined basis Zoo Foo is the minimal principal solution of (H) at 00. That is, W(Z^ Foo, ?oo) = 0, W^Foo, Zoo Yoq) = 0, and Z^Yqq = YoqK and ZqqYqq = FooK~l for some constant invertible n x n matrix K. Moreover, we have lim lu*(Zoo(t)Yoo(t),Yoo(t))=0, lim /x* (Z"1 (t) Y^ (t), Z"1 (t) E) = n - doo. (4.14) t^-oo t^oo Proof. By Remark 3.7, for the given transformation matrix R(t) := Zoo(0 there exists an an-tiprincipal solution F of (H) at 00, which is transformed into a maximal antiprincipal solution Y := Z^1 F of (H) at 00. Then the constant Wronskian W(ioo, F) is invertible, so that by (3.26) the Wronskian W(C„ a , Coo) is invertible as well. Hence, without loss of gen-erality (upon multiplying F by a constant invertible multiple) we can choose F such that W^Foo, Y) = W(C a , Coo) = /• Then by Proposition 4.3 the matrix Voo(t) defined in (4.6), which is associated with any such a conjoined basis F, satisfies lirn^oo hid [—Too (01 = 0. Moreover, by (3.49) in Corollary 3.11 (with R := Zoo) we get lim At*(0 ?oo(0, Y^t)) = lim ind[-Poo(0] - hid {[ WiY^, F)]"1 WiY^, Z^ Y^)} t^oo t^oo 1 1 = - ind {[ W(Foo, F)]"1 W(Foo, Zoo ?oo)}■ Since the left-hand side above is nonnegative and the right-hand side is nonpositive, it follows that both sides are zero, that is, lim Ai*(Zoo(0?oo(0,J>oo(0) = 0 = ind{[W(Foo,F)]-1 W(Foo,ZooFoo)}. (4.15) t^oo 1 ' This shows that the first equality in (4.14) holds. Applying equality (3.25) in Theorem 3.6 (with R := Zoo) together with (4.15) we obtain that ind {W(Foo, Zoo Yoo) (D - D^zJ [ WiY^, Z^ Y^)]7} = 0 (4.16) holds for every symmetric matrix D satisfying Im{W(Foo, Zoo ^oo ) (D - Doo zj tW^ Zoo Yoo)f} = ImW(Foo, Zoo ^00 )• (4.17) Then the symmetric matrix D := £) a — I satisfies (4.17), and hence by (4.16) we have P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2943 0 = ind{-W(Foo, Zoo Fx) )[ W(Foo, Zoo F0O)]r }= rank W(Foo, Zoo F>o). This implies that W(Z^ Fx, Fx) = Fx, Zoo Fx) = 0. Hence, Theorem 3.8 yields that Zoo1 ^oo iS the minimal principal solution of (H) at oo and Zoo Fx is the minimal principal solution of (H) at oo. Finally, we prove the second equality in (4.14) by applying Remark 3.4 with R(t) := Zoo(0- The limit in (3.18) is equal to zero. Then by (3.15) with W^Fx, Zoo Fx) = 0 and M(t) := Xoo(t) in combination with (3.17) it follows that lim ^(Z^COIooCO.Z"1^)^) = lim rank Zoo (0 = « -doo. The proof is complete. □ Remark 4.5. The result in Theorem 3.6 (respectively in Theorem 3.8) describes the situation when maximal antiprincipal solutions of (H) at oo are transformed into maximal antiprinci-pal solutions of (H) at oo under a general symplectic transformation matrix R(t). On the other hand, the result in Theorem 4.4 shows that the minimal principal solution of (H) at oo is transformed into the minimal principal solution of (H) at oo under the special transformation matrix R(t) := Zoo(0 and when (1.1) holds. This poses an interesting open problem regarding the general situation, when principal (antiprincipal) solutions of (H) at oo would be transformed into some principal (antiprincipal) solutions of (H) at oo. In the controllable case these results are known in [10, Theorems 1 and 2]. As the final result of this section we derive limit results for the comparative indices in (3.4) with R(t) := Zoo(0 by applying Theorem 3.2 to this case. Theorem 4.6. Assume that (1.1) and (1.2) hold on X = [a, oo) and that system (H) is nonoscil-latory at oo. Let Zoo be the symplectic fundamental matrix of(H) such that Foo = Zoo E. Then for every conjoined basis Y of (H) the limits lim /x(F(f),Foo(0) and lim /x*(F(f), £»(0) (4.18) exist and satisfy lim fi(Y(t), Y00(t)) = fhL(oo)-mL(oo)+mLoo(oo), (4.19) t^oo lim/x*(F(0,Foo(0) = 0, (4.20) t^oo where m^(oo) is the multiplicity of a focal point at oo of the conjoined basis Y := Z^1 F of the transformed system (H). Proof. By Proposition 4.1 we know that systems (H) and (H) are nonoscillatory at oo. Then the matrix M(t) := Xoo(0 has eventually constant kernel, and hence condition (3.9) is satisfied in this case. Therefore, by Theorem 3.2 (with R(t) := Zoo(0) the limits in (4.18) exist. By Theorem 4.4 we have WXFx, Zoo ?oo) = 0, so that by (2.23) the representation matrix C a has its first component zero. Then we have /x(Coo, ? ) = 0 = /x*(Coo, CL, ? ) by (2.6) and (2.7). Therefore, by using (3.11) and the second equality in (4.14) we get 2944 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 lim [i(Y(t), Foo(O) = mL(oo) -mL(oo) + n - d^ = mL(oo) -mL(oo) + mLoo(cc), t^oo where we used that m^^oo) = n — doo, see Section 2. This proves (4.19). Similarly, by using (3.12) and the first equality in (4.14) we get (4.20). The proof is complete. □ Remark 4.7. Equation (4.19) is an extension of formula (1.10) in Proposition 1.2 with Y := YtQ to the case to = oo. At the same time it is a generalization of formula (3.5) in Proposition 3.1 with Y* := Yoo to two systems (H) and (H) satisfying (1.1). Note that equation (4.20) represents an extension to to = oo of the left continuity of the dual comparative index /x*(Y(t), Y0(t)) at the point to in Proposition 1.3. 5. Singular Sturmian comparison theorems In this section we present the Sturmian comparison theorems for two linear Hamiltonian systems (H) and (H) on 1 = [a, oo) satisfying the majorant condition (1.1). In particular, we generalize the comparison theorems in Propositions 1.1, 1.2, and 1.3, as well as the separation theorems in Propositions 2.4 and 2.6 to this case. At the same time we do not assume any controllability condition on systems (H) and (H). The above assumptions imply (by Proposition 4.1) that the systems (H), (H), and (H) (with R(t) := Z(t) being a symplectic fundamental matrix of system (H)) are nonoscillatory at oo. Therefore, their conjoined bases have finitely many left and right focal points in [a, oo), and hence counting and comparing these numbers over unbounded intervals makes sense. Specifically, we will use the special transformation matrix R(t) := Zoo(f), as in Section 4. The main results are formulated by using the notation in (1.5) for the numbers of left and right focal points of conjoined bases of (H), (H), (H) in the interval 1 including the multiplicities of focal points at oo. The most general result in this section reads as follows. Theorem 5.1 (Singular Sturmian comparison theorem). Assume that (1.1) and (1.2) hold on I =[a, oo) and that system (H) is nonoscillatory at oo. Let be the symplectic fundamental matrix of (H) such that Y^ = Z^E. Then for every conjoined basis Y of (H) and for every conjoined basis Y of (H) we have m^(a, oo] — m^(a, oo] = m^(a, oo] + /x(F(a), Foo(o)) — R(Y(a), Yoo(«))> (5.1) mR[a, oo) — m«[a, oo) = m«[a, oo) + /x*(F(a), Foo(o)) — R*(Y(a), Foo(o)). (5.2) where rh^ia, oo] and rriR[a, oo) are the numbers of left and right focal points of the conjoined basis Y := Z^1 Y of (H) in the indicated intervals. Remark 5.2. (i) When systems (H) and (H) coincide (see also Remark 3.5), then for every conjoined basis Y of (H) we have m^(a, oo] = 0 = m«[a, oo). In this case Theorem 5.1 reduces to Proposition 2.4. (ii) In order to derive a suitable generalization of Proposition 2.4 to two systems (H) and (H), it is essential to understand that the differences m^(a, oo] — m^(a, oo] and m«[a, oo) — m#[a, oo) in Proposition 2.4 are expressed as (2.25) and (2.27). (hi) In Propositions 1.2 and 1.3 the transformed system (H) changes with the chosen conjoined basis Y of (H). The formulation in Theorem 5.1 has the advantage that the transformed system P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2945 (H) is the same for all conjoined bases Y of (H), since we use the special transformation matrix R(t) := Zoo(f), which does not depend on Y. The proof of Theorem 5.1 is presented below after the following special result regarding the case of Y := Yoo- It is a generalization of the second equality in (2.28) and of the first equality in (2.29) in Proposition 2.6 to two systems (H) and (H). At the same time it is extensions of (1.11) and (1.13) to the case of b = oo with the fundamental matrix Z(t) := Zoo(t). Theorem 5.3 (Singular Sturmian comparison theorem). Assume that (1.1) and (1.2) hold on I = [a, oo) and that system (H) is nonoscillatory at oo. Let Zoo be the symplectic fundamental matrix of(YL) such that Yoo = Zoo E. Then for every conjoined basis Y of (H) we have mL(a, oo] = inLoo(a, oo] + mL(a, oo] - /x(F(a), Foo(«))> (5.3) mR[a, oo) = mRoo[a, oo) oo) + /x*(F(a), Y00(a)), (5.4) where nii(a, 00] and mR[a, 00) are the numbers of left and right focal points of the conjoined basis Y := Z^1 Y of(H) in the indicated intervals. Proof. The proof of Theorem 5.3 is based on a combination of Propositions 1.2 and 1.3 (with the special choice Y := Yoo) and Theorem 4.6. By taking the limit for b —>- 00 in formula (1.11) in Proposition 1.2 we obtain mL(a, 00) = mLoo(a, 00) + mL(a, 00) + lim [i(Y(b), Y00(b)) - /x(F(a), Y00(a)) b^-oo (4 19) ^ ^ = mLoo(a, 00] +mL(a, 00] - mL(oo) - /x(F(a), Y00(a)). Hence, equality (5.3) follows. Similarly, by taking the limit for b 00 in formula (1.13) in Proposition 1.3 we get mR[a, 00) = m~Roo[a, 00) +mR[a, 00) + /x*(F(a), Y00(a)) - lim ix*(Y(b),Y00(b)) b^-oo (4=0) mRoo[a, 00) + mR[a, 00) + /x*(F(a), Y00(a)), which proves equality (5.4). □ The results in Theorem 5.1 now follow from Theorem 5.3 and Proposition 2.6. Proof of Theorem 5.1. Let F and Y be conjoined bases of (H) and (H), respectively. By the second equality in (2.28) in Proposition 2.6 applied to Y and Yoo, being conjoined based of system (H), we obtain that m^(a, 00] = m,Loo(a, 00] — /x(F(a), Yoo (a)). Using this equality in (5.3) yields equation (5.1). Similarly, the first equality in (2.29) in Proposition 2.6 applied to Y and Yoo implies that mR[a, 00) = mRoo[a, 00) + /x*(F(a), Yoo(a)), which together with (5.4) yields equation (5.2). The proof is complete. □ The results in Theorem 5.1 (or Theorem 5.3) allow to derive various estimates for the numbers of left and right focal points of conjoined bases of (H) and (H). Our first result shows the exact 2946 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 relationship between the numbers of focal points of the (minimal) principal solutions F>o, Foo, Foo and Ya,Ya, Ya of systems (H), (H), (H). Corollary 5.4. Assume that (1.1) and (1.2) hold onX = [a, oo) and that system (H) is nonoscillatory at oo. Then we have mLoo(a, oo] = mLoo(a, oo] + mLoo(a, oo] - /x(Y00(a), Foo (a)), (5.5) mRoo[a, oo) = mRoo[a, oo) + mRoo[a, oo) + /x*(Y00(a), Foo (a)), (5.6) mLa(a, oo] = mLa(a, oo] + mLa(a, oo] + /x*(Y00(a), Foo(a)), (5.7) mRa[a, oo) = mRa[a, oo) + mRa[a, oo) - /x(Foo(a), Foo(a)), (5.8) Proof. Equalities (5.5) and (5.6) follow from Theorem 5.3 with F := Fxd. Indeed, in this case the conjoined basis F from Theorem 5.3 satisfies F = Z^1 Fx> = Foo, by Theorem 4.4. Equalities (5.7) and (5.8) then follow directly from (5.6) and (5.5) with the aid of (2.33) for systems (H), (H), and (H). □ The following result confirms the intuitively expected fact that, given the same initial conditions, conjoined bases of the majorant system (H) have in general more focal points than conjoined bases of the minorant system (H). It is a generalization of the first part of [22, Corollary 7.3.2, pg. 196]. Theorem 5.5 (Singular Sturmian comparison theorem). Assume that (1.1) and (1.2) hold on I =[a, oo) and that system (H) is nonoscillatory at oo. Let Zoo be the symplectic fundamental matrix of (H) such that Foo = Zoo E. Let Y and Y be conjoined bases of(R) and (H), respectively, such that Y(a) = Y(a) K for some invertible matrix K. Then m^(a, oo] — m^(a, oo] = m^(a, oo] > 0, (5.9) mR[a, oo) — mR[a, oo) =mR[a, oo) > 0, (5.10) where m^a, oo] and mR[a, oo) are the numbers of left and right focal points of the conjoined basis Y := Z^1 F of (H) in the indicated intervals. Proof. This result follows directly from Theorem 5.1, in which we realize that the equalities /x(F(a), Foo(a)) = /x(F(a), Foo(a)) and /x*(F(a), Foo(a)) = /x*(F(a), Foo(a)) hold due to the assumption Y(a) = Y(a)K and property (2.7) of the comparative index. □ Remark 5.6. The equalities in (5.9) and (5.10) also provide the information about the relation between the numbers of focal points of the conjoined bases F and F := Z^1 F. Namely, we have mi(a, oo] > rhi,ia, oo] and mR[a, oo) >mR[a, oo). Next we compare the numbers of focal points of the (minimal) principal solutions Ya, Ya, Ya and F>o, Foo, F^. We also provide universal lower and upper bounds for the numbers of focal points of conjoined bases of (H) and (H). P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2947 Corollary 5.7. Assume that (1.1) and (1.2) hold on I =[a, oo) and that system (H) is nonoscil-latory at oo. Then for every conjoined basis Y of (H) the estimates in (2.30) and (2.31) hold, where the lower and upper bounds satisfy inLa(a,oc] +inLa(a,oc] m^ooia, oo] by Theorem 5.5. At the same time, for the conjoined bases Y and Fx, of (H) we have the inequality m^{a, oo] < mLOQ{a, oo], by (2.30). Therefore, m-L^ia, oo] < mLOQ{a, oo] holds, which completes the proof of (5.13). Finally, since Ya(a) = E = Ya(a) holds, then the third inequality in (5.14) follows from Theorem 5.5 (with Y := Ya and Y := Ya). The proof is complete. □ The estimates in Corollary 5.7 show that the proper generalization of Proposition 1.1 to uncontrollable systems should be done through the right focal points. At the same time Corollary 5.7 provides an extension of Proposition 1.1 to the left focal points. Remark 5.8. The results of this paper show the importance of the transformed system (H) for comparing the numbers of focal points of conjoined bases of systems (H) and (H). Therefore, the transformed system (H) can be considered as a quantitative measure of the majorant condition (1.1). For example, the optimal upper bounds wiLoo(fl, oo] — mRa\a, oo) and m^^ia, oo] — mRa[a, oo) for the numbers of left and right focal points of conjoined bases Y and Y of (H) and (H) satisfy the estimates 0 < mLoo(a, oo] - inLoo(a, oo] < inLoo(a, oo], 0 < mRa[a, oo) - inRa[a, oo) < inRa[a, oo). We believe that further investigation of the properties of the transformed system (H) will lead to better explanation of the role of condition (1.1) in the Sturmian theory of these systems. Another open problem is to understand how the maximal orders of abnormality doo, doo, doo of systems (H), (H), (H) affect the transformation rules for principal and antiprincipal solutions of (H) at oo. We will address these issues in our subsequent work. 2948 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 6. Further Sturmian comparison theorems on unbounded intervals The results of this paper extend in analogous way to the unbounded intervals of the form (—oo, b], where for measuring the numbers of focal points we use the corresponding (minimal) principal solutions Y-oo, f-oo, F-oo and Yb, fb, fb- For the definition of a principal solution of (H) at — oo we refer to [33, Section 2]. The corresponding Sturmian separation theorems on the intervals of the form (—oo,b] and the limit results for the comparative index at — oo for conjoined bases of one system (H) are discussed in [33, Remarks 5.16 and 6.5]. For completeness and future reference we provide the statements for two systems (H) and (H) below. Consider the linear Hamiltonian systems (H) and (H), as well as the transformed system (H), on the unbounded interval X = (—oo, b]. We assume that they satisfy majorant condition (1.1) and the Legendre condition (1.2) on this interval, which in turn implies the validity of the Legen-dre conditions (2.13) and (3.3) on (—oo, b]. If Y is a conjoined basis of (H) with constant kernel on (—oo, ft] for some ft e (—oo, b] with d(—oo, ft] = d-oo, then analogously to (2.19) we define the symmetric matrix f T^-00:=tnmo^JxHs)B(s)XfT(s)ds), 0 < rank7>, < n - d-oo, (6.1) compare with [32, Section 5]. Note that Tp, -oo < 0. Following [33, Remark 3.6] we then define the multiplicity of the (right) proper focal point of Y at — oo by mR(—oo):=n — d-O0—T&akTp-O0, 0 < niR(—oo) < n — d-oo. (6.2) In this case m^(—oo) = 0 if and only if Y is an antiprincipal solution of (H) at —oo (i.e., rank 7^-00 = n — d-oo), while m^(-oo) = n — d-oo if and only if Y is a principal solution of (H) at —oo (i.e., Tp, -oo = 0). Furthermore, as in (2.21) we have the formula mR(-oo)= lim rankX(f)-rankWr(F_00, T), (6.3) t^—oo where Y-oo is the minimal principal solution of (H) at —oo. We begin with an analogue of Proposition 4.1. Proposition 6.1. Assume that (1.1) and (1.2) hold on X = (—oo,b] and that system (H) is nonoscillatory at — oo. Then (i) system (H) is nonoscillatory at — oo, and (ii) for every symplectic fundamental matrix Z of(H) the transformed system (H) with the coefficient matrix H(t) given in (1.14) is nonoscillatory at —oo. Next we present an analogue of Theorem 4.6. Observe that formula (6.5) below is an extension of (1.12) to the case of to = —oo. Theorem 6.2. Assume that (1.1) and (1.2) hold on the interval X = (—oo, b] and that system (H) is nonoscillatory at — oo. Let Z-oo be the symplectic fundamental matrix of (H) such that f-oo = Z-oo E. Then for every conjoined basis Y of (H) the limits P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2949 lim fi(Y(t),Y-oo(t)) and lim fi*(Y(t), F-oo(O) t^— 00 t^—oo exist and satisfy lim /x(F(f),F-oo(0)=0, (6.4) t^—oo lim /x*(F(f), F-oo(O) = mR(-oo) -mR{-oo) + inR-00(-oo), (6.5) f^— 00 where m R{—oo) is the multiplicity of a focal point at —oo of the conjoined basis Y := ZZ,^, Y of the transformed system (H). Next we present a transformation result for the minimal principal solutions at — oo and for the maximal antiprincipal solutions at —oo. It is an analogue of Theorems 3.8 and 4.4. Theorem 6.3. Assume that (1.1) and (1.2) hold on the interval X = (—oo, b] and that system (H) is nonoscillatory at — oo. Let Z_oo be the symplectic fundamental matrix of (H) such that F_oo = Z_oo E. Then the following statements hold. (i) The conjoined basis ZZ,^, F-oo is the minimal principal solution of (H) at —oo and the conjoined basis Z_oo Y-oq is the minimal principal solution of (H) a? —oo. (ii) Every maximal antiprincipal solution of(H) at — oo w transformed into a maximal antiprincipal solution of(H) at —oo under the transformation Y = ZZ^CO Y. (iii) Every maximal antiprincipal solution of(H) at — oo is a transformation of some maximal antiprincipal solution of (H) at — oo under Y = ZZ^CO ^- The results in Theorems 5.1 and 5.3 have the following counterpart. Observe that formulas (6.8) and (6.9) below are extensions of (1.11) and (1.13) to the case of a = —oo with the fundamental matrix Z(t) := Z_oo(f). Theorem 6.4 (Singular Sturmian comparison theorem). Assume that (1.1) and (1.2) hold on I = (—oo, b] and that system (H) is nonoscillatory at —oo. Let Z_oo be the symplectic fundamental matrix of (H) such that F_oo = Z_oo E. Then for every conjoined basis Y of (H) and for every conjoined basis Y of (H) we /zave mL(-oo, fc] - mL(-oo, fc] = mL(-oo, fc] + /x(F(6), F-ooW) - fi(Y(b), F-ooW), (6-6) mj?[-oo, fc) - mR[-oo, b) = mR[-oo, b) + fi*(Y(b), Y-^b)) - fi*(Y(b), F-ooW), (6.7) where m^ (—oo, Z?] an mi-ooi-oo, oo) > ^-^(-oo, oo) + ini-ooi-oo, oo), (6.18) mR(—oo, oo) > mRoo(—oo, oo) > inRoo(—oo, oo) + mRoo(—oo, oo). (6.19) The inequalities in (6.18) follow from (6.10) with b oo by dropping the last term with the comparative index, while the inequalities in (6.19) follow from (5.6) with a —oo by dropping the last term with the dual comparative index. These lower bounds for the numbers of left and right focal points of Y in (—oo, oo) improve the lower bound w700(—oo, oo) obtained in (1.9) of Proposition 1.1 for a — oo. Remark 6.7. Unlike in [33, Remark 8.1] for the Sturmian separation theorems, the results in Section 5 on 1 = [a, oo) together with the results in this section on 1 = (—oo, b] do not combine in general to Sturmian comparison theorems on the entire interval 1 = (—oo, oo). The main reason is that we employ two different transformation matrices R(t) = Z±oo (t) in neighborhoods of ±oo, which yield two different transformation systems (H). Therefore, the question of the validity of the Sturmian comparison theorems for systems (H) and (H) on the unbounded interval X = (—oo, oo) remains an open problem when the minimal principal solutions Foo and F_oo of (H) at ±oo differ, meaning that F_oo is not a constant nonsingular multiple of Foo- In other words, the results presented in Sections 4 and 5 (with a = — oo) and in Section 6 (with b = oo) remain valid under the additional assumption that Foo = F_oo (i.e., Zoo = Z_oo). Following the discussion in Remark 6.7 we present below the extensions of Theorem 6.4 and Corollary 6.5 to the case of X = (—oo, oo) under the additional assumption Foo = F_oo- P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2951 Theorem 6.8 (Singular Sturmian comparison theorem). Assume that (1.1) and (1.2) hold onX = (—cc, oo), system (H) is nonoscillatory at ±oo, and that F_oo = Too. Let the symplectic fundamental matrix of (H) such that Too = Z^ E. 77?en /or every conjoined basis Y of (H) and for every conjoined basis Y o/(H) we /zave mi(—oo, oo] — m^—oo, oo] = m^—oo, oo], (6.20) m#[—oo, oo) — m«[—oo, oo) = mR[—oo, oo), (6.21) where nii(—oo, oo] and m#[—oo, oo) are the numbers of left and right focal points of the conjoined basis Y := Z^,1 Y of(&) in the indicated intervals. In particular, >«l-oo(-°o, oo] = mL_oo(—oo, oo] + ntL-^—oo, oo], (6.22) niR-voi—OQ, oo) = m^-oof-oo, oo) + m/j_oo[—oo, oo), (6.23) mLoo(-oo, oo] = mLoo(-oo, oo] + mLoo(-oo, oo], (6.24) mR(Xl[—OQ, oo) = mRoo[-oo, oo) + mRoo[-oo, oo), (6.25) and mL_oo(-oo, oo] > mL_oo(—oo, oo], mSoo[-oo, oo) > mSoo[-oo, oo). (6.26) Proof. Let F and F be conjoined bases of (H) and (H), respectively. Under the given assumptions we fix an arbitrary point aeK. Then Theorem 5.1 yields that (5.1) and (5.2) hold, where Y := Z^1 F. Next, from Theorem 6.4 (with b := a) we obtain m^(—oo, a] — m^(—oo, a] = m^(—oo, a] + /x(F(a), F_oo(a)) — /x(F(a), F_oo(a)), (6.27) mR[—oo, a) — m.R\—oo, a) = oo, a) + /x*(F(a), F_oo(a)) — /x*(F(a), F_oo(a)), (6.28) where F := ZZ™ F. Since we now assume that Y—qq = Yqq, then we may take Z—qq = Zqq, so that the conjoined basis Y in (5.1) and (5.2) coincides with the conjoined basis Y in (6.27) and (6.28). Upon adding equation (5.1) with (6.27), and equation (5.2) with (6.28), and using that F_oo = Foo holds, we obtain the equalities in (6.20) and (6.21). In particular, for F := F_oo and Y := F_oo we have Y = ZZ^o F_oo = F-oo by Theorem 6.3, so that equality (6.22) follows from (6.20), and equality (6.23) follows from (6.21). Similarly, equalities (6.24) and (6.25) follow from (6.20) and (6.21) with F := F^ and Y := F^ (noting that Y = Z"1 F^ = by Theorem 4.4). Finally, the estimates in (6.26) are obtained from (6.22) and (6.25) by dropping the last term in the sum on the right-hand side. □ Remark 6.9. As we mentioned in Section 1, the results of this paper are new even for the case of "weakly disconjugate" linear Hamiltonian systems (H) and (H) at oo, resp. at —oo, i.e., for systems with doo = 0 = doo, resp. d-oo = 0 = d-oo. (6.29) 2952 P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 In particular, the results are new for completely controllable systems (H) and (H), which automatically satisfy condition (6.29). Among the latter ones we mention the second order Sturm-Liouville differential equations. We will present consequences of our new theory for this special case in a separate study. Appendix A. Auxiliary results about normalized conjoined bases In this section we present a completion of some known results from matrix analysis related to normalized conjoined bases. The basic problem is to find for a given constant 2nxn matrix F satisfying YTJY = 0 and rank Y = n (called for simplicity also a conjoined basis) another constant 2nxn matrix Y satisfying the same properties such that their Wronskian matrix W(Y, Y) = I. In this case we say that the matrices Y and Y are normalized. A classification of all such conjoined bases Y with W(Y, Y) = I is derived in [22, Corollary 3.3.9], where it is also stated that the conjoined basis Y may be chosen so that the first component X := (I, 0) Y is invertible, see also [22, Proposition 4.1.1]. In this section we complete this result by deriving a classification of all such conjoined bases Y with X invertible. This result is then used in the proof of Theorem 3.6, which is the main tool for the transformation theory of principal and antiprincipal solutions at oo (Theorems 3.8 and 4.4). In accordance with (2.14) we split Y and Y into their n x n components Y = (XT, U7)7 and Y = (X7, U7)7. Then W(Y, Y) = I if and only if YYT - YYT =J, that is, XUT-XUT = I, XXT and UUT are symmetric, (A.l) see [22, Proposition 1.1.5]. Moreover, we define the n x n matrices F:=YTY>0, G:=F_1/2>0, H := GXT(GXT)\ (A.2) DY :=GHGUT(GXT)f G. (A3) We note that the matrix H is the orthogonal projector onto the subspace Im(GXT). Lemma A.l. Let Y be a conjoined basis with F, G, H, and Dy defined in (A.2) and (A3). Then the matrix Dy is symmetric and satisfies XDyXT = UF~lXT. Proof. First we note that the matrix Y := —JYF~l is a conjoined basis satisfying W(Y, Y) = I. Thenby(A.l)thematrixXF_1£/r = —XXT is symmetric. The symmetry of the matrix Dy then follows by DY = [(GXT)f G]7 XF-lUT[(GXT)^ G]. Moreover, since XGH = XG and G2 = F~l, it follows that XDYXT = XF~lU7\GXT)f GXT = UGGXT(GXT)f GXT = UGGX7 = UF~lXT, which completes the proof. □ Next we present the main result of this section. P. Šepitka, R. Šimon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2953 Theorem A.2. Let Y be a conjoined basis with F and Dy defined in (A.2) and (A.3). Then a matrix Y is a conjoined basis with W(Y, Y) = I and with the matrix X invertible if and only if Y = —JYF~l + YD, where D is a symmetric n x n matrix satisfying lm[X(D - DY)XT] = lmX. (A.4) In this case we have ind[X(D - DY)XT] = md(X-1X). (A.5) Remark A.3. (i) Condition (A.4) is equivalent with the inclusion ImX c lm[X(D — Dy) XT], or with Ker [X(D — Dy) XT] c KerXr, since the opposite inclusions hold trivially. (ii) We note that for a given conjoined basis Y there always exists a symmetric matrix D satisfying equality (A.4), e.g. D := Dy + X^X^T, compare also with [22, Corollary 3.3.9]. Hence, there always exists a conjoined basis Y with W(Y, Y) = I and X invertible. Proof of Theorem A.2. By [22, Corollary 3.3.9], every conjoined basis Y with W(Y, Y) = I is of the form Y = —JYF~l + YD with a symmetric n x n matrix D. In particular, the corresponding matrix X has the form X = — UF~l +XD. For a given D we will show that KerX = XT Ker[X(D - DY)XT]. (A.6) Let v e KerX. Since W(Y, Y) = I, we have the formula XTU — UTX = I and hence v = (XTU — UTX) v = XTw with w := Uv. Moreover, by Lemma A.l, X(D - Dy) XTw = (XDXT - UF~lXT) w = XXTw = Xv = 0. Therefore, we have w g Ker [X(D - DY) XT] and d = XTw g XT Ker [X(D - Dy) XT]. Hence, the inclusion KerX c XT Ker [X (D — Dy) XT] holds. Conversely, assume that v = XTw with w g Ker [X(D - DY)XT]. With the aid of Lemma A.l we then get Xv = (XD - UF~l) XTw = (XDXT - XDYXT) w = X(D - Dy) XTw = 0, i.e., the vector v e KerX. Thus, also the inclusion XT Ker[X(D — Dy)XT] c KerX holds and (A.6) is proven. Finally, it is easy to see from (A.6) that the matrix X is nonsingular, i.e., the subspace KerX = {0}, if and only if Ker[X(D - DY)XT] c KerXr. According to Remark A.3(i), the last inclusion is equivalent with equality (A.4). Finally, by the above expression of it = -UF~l + XD and by Lemma A.l we obtain XXT = X(D - DY)XT, which implies (A.5), since X is invertible. The proof is complete. □ References [1] D. Aharonov, U. Elias, Singular Sturm comparison theorems, J. Math. Anal. Appl. 371 (2) (2010) 759-763. [2] W.O. Amrein, A.M. Hinz, D.B. Pearson, Sturm-Liouville theory. Past and present, including papers from the International Colloquium held at the University of Geneva, Geneva, 2003, Birkhäuser Verlag, Basel, 2005. [3] V.I. Arnold, Sturm theorems and symplectic geometry, Funkc. Anal. Prilozh. 19 (4) (1985) 1-10; translated in Funct. Anal. Appl. 19 (4) (1985) 251-259. [4] F.V. Atkinson, Discrete and Continuous Boundary Problems, Academic Press, New York-London, 1964. 2954 P. Šepitka, R. Šimon Hilscher / J. Differential Equations 269 (2020) 2920-2955 [5] A. Ben-Israel, T.N.E. Greville, Generalized Inverses: Theory and Applications, second edition, Springer-Verlag, New York, NY, 2003. [6] D.S. Bernstein, Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory, Princeton University Press, Princeton, 2005. [7] S.L. Campbell, CD. Meyer, Generalized Inverses of Linear Transformations, reprint of the 1991 corrected reprint of the 1979 original, Classics in Applied Mathematics, vol. 56, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2009. [8] W.A. Coppel, Comparison theorems for canonical systems of differential equations, J. Math. Anal. Appl. 12 (2) (1965) 306-315. [9] W.A. Coppel, Disconjugacy, Lecture Notes in Mathematics, vol. 220, Springer-Verlag, Berlin - New York, 1971. [10] O. Došlý, Principal solutions and transformations of linear Hamiltonian systems, Arch. Math. (Brno) 28 (1992) 113-120. [11] O. Došlý, Relative oscillation of linear Hamiltonian differential systems, Math. Nachr. 290 (14-15) (2017) 2234-2246. [12] O. Došlý, W. Kratz, Singular Sturmian theory for linear Hamiltonian differential systems, Appl. Math. Lett. 26 (12) (2013)1187-1191. [13] JV. Elyseeva, The comparative index for conjoined bases of symplectic difference systems, in: S. Elaydi, J. Cush-ing, R. Lasser, A. Ruffing, V. Papageorgiou, W. Van Assche (Eds.), Difference Equations, Special Functions, and Orthogonal Polynomials, Proceedings of the International Conference, Munich, 2005, World Scientific, London, 2007, pp. 168-177. [14] JV. Elyseeva, Comparative index for solutions of symplectic difference systems, Differ. Equ. 45 (3) (2009) 445^-59; translated from Differ. Uravn. 45 (3) (2009) 431^144. [15] JV. Elyseeva, Comparison theorems for conjoined bases of linear Hamiltonian differential systems and the comparative index, J. Math. Anal. Appl. 444 (2) (2016) 1260-1273. [16] JV. Elyseeva, On symplectic transformations of linear Hamiltonian differential systems without normality, Appl. Math. Lett. 68 (2017) 33-39. [17] JV. Elyseeva, The comparative index and transformations of linear Hamiltonian differential systems, Appl. Math. Comput. 330 (2018) 185-200. [18] R. Fabbri, R. Johnson, C. Nunez, On the Yakubovich frequency theorem for linear non-autonomous control processes, Discrete Contin. Dyn. Syst. 9 (3) (2003) 677-704. [19] P. Hartman, Ordinary Differential Equations, John Wiley, New York, 1964. [20] D.A. Harville, Matrix Algebra from a Statistician's Perspective, Springer-Verlag, New York, 1997. [21] R. Johnson, R. Obaya, S. Novo, C. Núňez, R. Fabbri, Nonautonomous Linear Hamiltonian Systems: Oscillation, Spectral Theory and Control, Developments in Mathematics, vol. 36, Springer, Cham, 2016. [22] W. Kratz, Quadratic Functionals in Variational Analysis and Control Theory, Akademie Verlag, Berlin, 1995. [23] W. Kratz, Definiteness of quadratic functionals, Analysis 23 (2) (2003) 163-183. [24] W. Kratz, R. Simon Hilscher, Rayleigh principle for linear Hamiltonian systems without controllability, ESAIM Control Optim. Calc. Var. 18 (2) (2012) 501-519. [25] W.T. Reid, Ordinary Differential Equations, John Wiley & Sons, Inc., New York - London - Sydney, 1971. [26] W.T. Reid, Sturmian Theory for Ordinary Differential Equations, Springer-Verlag, New York - Berlin - Heidelberg, 1980. [27] FS. Rofe-Beketov, A.M. Kholkin, Spectral Analysis of Differential Operators, translated from the Russian by Ogn-jen Milatovic and revised by the authors. With a foreword by Vladimir A. Marchenko, World Scientific Monograph Series in Mathematics, vol. 7, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2005. [28] P. Sepitka, R. Simon Hilscher, Minimal principal solution at infinity for nonoscillatory linear Hamiltonian systems, J. Dyn. Differ. Equ. 26 (1) (2014) 57-91. [29] P. Sepitka, R. Simon Hilscher, Principal solutions at infinity of given ranks for nonoscillatory linear Hamiltonian systems, J. Dyn. Differ. Equ. 27 (1) (2015) 137-175. [30] P. Sepitka, R. Simon Hilscher, Principal and antiprincipal solutions at infinity of linear Hamiltonian systems, J. Differ. Equ. 259 (9) (2015) 4651^1682. [31] P. Sepitka, R. Simon Hilscher, Comparative index and Sturmian theory for linear Hamiltonian systems, J. Differ. Equ. 262 (2) (2017) 914-944. [32] P. Sepitka, R. Simon Hilscher, Focal points and principal solutions of linear Hamiltonian systems revisited, J. Differ. Equ. 264 (9) (2018) 5541-5576. [33] P. Sepitka, R. Simon Hilscher, Singular Sturmian separation theorems on unbounded intervals for linear Hamiltonian systems, J. Differ. Equ. 266 (11) (2019) 7481-7524. P. Sepitka, R. Simon Hilscher / J. Differential Equations 269 (2020) 2920-2955 2955 [34] R. Simon Hilscher, Sturmian theory for linear Hamiltonian systems without controllability, Math. Nachr. 284 (7) (2011) 831-843. [35] R. Simon Hilscher, On general Sturmian theory for abnormal linear Hamiltonian systems, in: W. Feng, Z. Feng, M. Grasselli, A. Ibragimov, X. Lu, S. Siegmund, J. Voigt (Eds.), Dynamical Systems, Differential Equations and Applications, Proceedings of the 8th AIMS Conference on Dynamical Systems, Differential Equations and Applications, Dresden, 2010, in: Discrete Contin. Dynam. Systems, Suppl. 2011, American Institute of Mathematical Sciences (AIMS), Springfield, MO, 2011, pp. 684-691. [36] M. Wahrheit, Eigenvalue problems and oscillation of linear Hamiltonian systems, Int. J. Difference Equ. 2 (2) (2007) 221-244.