#### DOCUMENTA MATHEMATICA, Vol. 18 (2013), 785-855

Jan Möllers

A Geometric Quantization of the Kostant--Sekiguchi Correspondence for Scalar Type Unitary Highest Weight Representations

For any Hermitian Lie group $G$ of tube type we give a geometric quantization procedure of certain $K_\CC$-orbits in $\frakp_\CC^*$ to obtain all scalar type highest weight representations. Here $K_\CC$ is the complexification of a maximal compact subgroup $K\subseteq G$ with corresponding Cartan decomposition $\frakg=\frakk+\frakp$ of the Lie algebra of $G$. We explicitly realize every such representation $\pi$ on a Fock space consisting of square integrable holomorphic functions on its associated variety $\Ass(\pi)\subseteq\frakp_\CC^*$. The associated variety $\Ass(\pi)$ is the closure of a single nilpotent $K_\CC$-orbit $\calO^{K_\CC}\subseteq\frakp_\CC^*$ which corresponds by the Kostant--Sekiguchi correspondence to a nilpotent coadjoint $G$-orbit $\calO^G\subseteq\frakg^*$. The known Schrödinger model of $\pi$ is a realization on $L^2(\calO)$, where $\calO\subseteq\calO^G$ is a Lagrangian submanifold. We construct an intertwining operator from the Schrödinger model to the new Fock model, the generalized Segal--Bargmann transform, which gives a geometric quantization of the Kostant--Sekiguchi correspondence (a notion invented by Hilgert, Kobayashi, Ørsted and the author). The main tool in our construction are multivariable $I$- and $K$-Bessel functions on Jordan algebras which appear in the measure of $\calO^{K_\CC}$, as reproducing kernel of the Fock space and as integral kernel of the Segal--Bargmann transform. As a corollary to our construction we also obtain the integral kernel of the unitary inversion operator in the Schrödinger model in terms of a multivariable $J$-Bessel function as well as explicit Whittaker vectors.

2010 Mathematics Subject Classification: Primary 22E45; Secondary 17C30, 30H20, 33C70, 44A15, 46E22.

Keywords and Phrases: unitary highest weight representation, orbit method, Schrödinger model, Fock model, Jordan algebra, Segal–-Bargmann transform, Bessel function, Bessel operator, unitary inversion operator, Whittaker vectors, branching law.

Full text: dvi.gz 134 k, dvi 469 k, ps.gz 497 k, pdf 644 k.