The discovery of flavor neutrino oscillations (see Ref. [1] and the references therein) proves that neutrinos are massive. This leads to an important consequence that the lepton-flavor number violating decay μ→eγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu \rightarrow e \gamma $$\end{document} is non-vanishing, being proportional to the neutrino masses and the mixing matrix. Assuming tiny neutrino masses satisfying current experimental constraints [1], extension of the Standard Model (SM) with right-handed neutrinos predicts that the branching ratio is Br≈10-55\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Br} \approx 10^{-55}$$\end{document}, which will be called the SM contribution from now on. Meanwhile, the current experimental limits read [1]1Br(μ-→e-γ)<4.2×10-13,Br(τ-→e-γ)<3.3×10-8,Br(τ-→μ-γ)<4.4×10-8.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\text {Br}(\mu ^- \rightarrow e^-\gamma )< 4.2\times 10^{-13},\nonumber \\&\text {Br}(\tau ^- \rightarrow e^-\gamma )< 3.3\times 10^{-8},\nonumber \\&\text {Br}(\tau ^- \rightarrow \mu ^-\gamma ) < 4.4\times 10^{-8}. \end{aligned}$$\end{document}From theoretical side, the processes li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} are loop induced. Given that the SM contribution is strongly suppressed, they can be good places to look for new physics. In this paper, we consider a simple extension of the SM using the local gauge group of SU(3)C⊗SU(3)L⊗U(1)X\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_C\otimes SU(3)_L\otimes U(1)_X$$\end{document} (3-3-1) with new exotic leptons. The word exotic here means that they can have arbitrary electric charges and arbitrary masses. In this model, the electron (and similarly for muon and tauon) together with a neutrino and a new exotic lepton are in a triplet (or anti-triplet) representation of SU(3)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L$$\end{document}. In this work, we calculate both neutrino and exotic-lepton contributions, with special attention to the latter because the former is numerically suppressed as mentioned above.

We remark that 3-3-1 model is an active field of research and has a long history; see Ref. [2] and the references therein. In this work, we choose a general class of 3-3-1 models, which are similar to the models presented in Refs. [2–4] where new heavy leptons are introduced. However, there is an important difference: instead of fixing the electric charges of the new leptons to specific values being 0, +1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+\,1$$\end{document} or -1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-\,1$$\end{document}, we let them be arbitrary. We will then study the dependence of the li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} branching fractions on this arbitrary charge. That class of 3-3-1 models has been studied in many works; see e.g. [5, 6]. If we replace the new leptons with charge-conjugated partners of the SM leptons, we will have different 3-3-1 models with lepton-number violation; see e.g. Refs. [7–11]. The decays li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} in these models have been discussed in Refs. [12, 13]; see also the recent review [14] and the references therein. We do not discuss these types of models in this work, but rather focus on the case with exotic leptons.

In the general class of 3-3-1 models here considered, there is one important parameter usually called β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}, which together with X, the new charge corresponding to the group U(1)X\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U(1)_X$$\end{document}, define the electric-charge operator. The electric charges of new particles therefore depend on β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}. It has been known and widely accepted that β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document} is one of the most important parameters to classify 3-3-1 models.

Recently, new efforts were made using 3-3-1 models to understand tensions between experimental measurements and the SM results in B physics; see e.g. [15, 16]. Motivated by this work, we want to use 3-3-1 models to understand the li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} decays. Since the new leptons are assumed to be heavy, we expect large branching fractions. However, this is not totally obvious, because there are two contributions from gauge and Higgs sectors. Does a destructive interference effect occur?

The aim of this paper is manifold. First, we calculate the full and exact result for li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} partial decay widths for a general class of 3-3-1 models with arbitrary β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}. As a by product, we will perform numerical comparisons between the exact results (i.e. external lepton masses are kept) and approximate ones where external lepton masses are neglected. We note that approximate results have been almost exclusively used in the literature for the SM and many other models. We found this uncomfortable because the neutrino masses, which are much smaller than the lepton masses, are kept. We therefore want to know to what accuracy the approximate results valid, using the SM with arbitrary neutrino masses to answer this. As far as we know, this important point has never been addressed in the literature. We will also perform numerical studies for 3-3-1 model to see whether destructive interference effects occur and to see the dependence on β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}, gauge boson and Higss masses. To the best of our knowledge, this is the first study of li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} in 3-3-1 models with exotic leptons.

The paper is organized as follows. In the next section, we review the model and calculate the Feynman rules needed for li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} decays. We then summarize the main calculation steps and present analytical results in Sect. 3. Numerical results are discussed in Sect. 4. In Sect. 4.1 we perform comparisons between the approximate and exact results for the neutrino contribution. In Sect. 4.2 we present results for the exotic-lepton contribution. Conclusions are in Sect. 5. Finally, we provide Appendices A and B to complete the results of Sect. 3.

One important condition we require is that the 3-3-1 model has to match the SM at the energy of the EW scale, about250GeV\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$250{\,\text {GeV}}$$\end{document}. This means that the SU(3)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L$$\end{document} symmetry is valid at a higher energy scale and is spontaneously broken down to the SU(2)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(2)_L$$\end{document} symmetry using the Brout–Englert–Higgs mechanism. In order to match the fermion representation of the SM, the simplest choice is to assign fermions into triplets and anti-triplets of the SU(3)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L$$\end{document} group. However, this requires new fermions. In general, the electric charges of these new fermions are unkown. They, however, cannot be totally arbitrary because of the symmetry and of the matching condition with the SM. In most general terms, the electric-charge operator can be written as2Q=T3+βT8+X1,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Q = T_3 + \beta T_8 + X \mathbb {1}, \end{aligned}$$\end{document}where we have introduced the SU(3) generators T3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}, T8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_8$$\end{document}. Thus, the charge operator Q depends on two parameters β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document} and X. With this information, we can write down the lepton representation as follows. Left-handed leptons are assigned to anti-triplets and right-handed leptons to singlets:3LaL′=ea′-νa′Ea′L∼3∗,-12+β23,a=1,2,3,eaR′∼1,-1,νaR′∼1,0,EaR′∼1,-12+3β2.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&L'_{aL}=\left( \begin{array}{c} e'_a \\ -\nu '_{a} \\ E'_a \\ \end{array} \right) _L \sim \left( 3^*~, -\frac{1}{2}+\frac{\beta }{2\sqrt{3}}\right) , \quad a=1,2,3,\nonumber \\&e'_{aR}\sim \left( 1~, -1\right) , \quad \nu '_{aR}\sim \left( ~1~, 0\right) ,\nonumber \\&E'_{aR} \sim \left( ~1~, -\frac{1}{2}+\frac{\sqrt{3}\beta }{2}\right) . \end{aligned}$$\end{document}The model includes three RH neutrinos νaR′\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nu '_{aR}$$\end{document} and exotic leptons EL,R′a\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E'^a_{L,R}$$\end{document} which are much heavier than the normal leptons. The prime denotes flavor states to be distinguished with mass eigenstates introduced later. The numbers in the parentheses are to label the representation of SU(3)L⊗U(1)X\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L\otimes U(1)_X$$\end{document} group. For singlets, we have Q=X\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q=X$$\end{document} and hence the electric charges of the new leptons can be read off from the above information. The quark sector is not specified here since it is irrelevant to our present work.

We now discuss gauge and Higgs interactions. There are totally nine EW gauge bosons, included in the following covariant derivative:4Dμ≡∂μ-igTaWμa-igXXT9Xμ,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{\mu }\equiv \partial _{\mu }-i g T^a W^a_{\mu }-i g_X X T^9X_{\mu }, \end{aligned}$$\end{document}where T9=1/6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^9=\mathbb {1}/\sqrt{6}$$\end{document}, g and gX\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_X$$\end{document} are coupling constants corresponding to the two groups SU(3)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L$$\end{document} and U(1)X\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U(1)_X$$\end{document}, respectively. The matrix WaTa\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^aT^a$$\end{document}, where Ta=λa/2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^a =\lambda _a/2$$\end{document} corresponding to a triplet representation, can be written as5WμaTa=12Wμ3+13Wμ82Wμ+2Yμ+A2Wμ--Wμ3+13Wμ82Vμ+B2Yμ-A2Vμ-B-23Wμ8,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W^a_{\mu }T^a=\frac{1}{2}\left( \begin{array}{ccc} W^3_{\mu }+\frac{1}{\sqrt{3}} W^8_{\mu }&{} \sqrt{2}W^+_{\mu } &{} \sqrt{2}Y^{+A}_{\mu } \\ \sqrt{2}W^-_{\mu } &{} -W^3_{\mu }+\frac{1}{\sqrt{3}} W^8_{\mu } &{} \sqrt{2}V^{+B}_{\mu } \\ \sqrt{2}Y^{-A}_{\mu }&{} \sqrt{2}V^{-B}_{\mu } &{}-\frac{2}{\sqrt{3}} W^8_{\mu }\\ \end{array} \right) , \nonumber \\ \end{aligned}$$\end{document}where we have defined the mass eigenstates of the charged gauge bosons as6Wμ±=12Wμ1∓iWμ2,Yμ±A=12Wμ4∓iWμ5,Vμ±B=12Wμ6∓iWμ7.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W^{\pm }_{\mu }=\frac{1}{\sqrt{2}}\left( W^1_{\mu }\mp i W^2_{\mu }\right) ,\nonumber \\ Y^{\pm A}_{\mu }=\frac{1}{\sqrt{2}}\left( W^4_{\mu }\mp i W^5_{\mu }\right) ,\nonumber \\ V^{\pm B}_{\mu }=\frac{1}{\sqrt{2}}\left( W^6_{\mu }\mp i W^7_{\mu }\right) . \end{aligned}$$\end{document}From Eq. (2), the electric charges of the gauge bosons are calculated as7A=12+β32,B=-12+β32.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A=\frac{1}{2}+\beta \frac{\sqrt{3}}{2}, \quad B=-\frac{1}{2}+\beta \frac{\sqrt{3}}{2}. \end{aligned}$$\end{document}We note that B is also the electric charge of the new leptons Ea\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_a$$\end{document}.

To generate masses for gauge bosons and fermions, we need three scalar triplets. They are defined as8χ=χ+Aχ+Bχ0∼3,β3,ρ=ρ+ρ0ρ-B∼3,12-β23η=η0η-η-A∼3,-12-β23,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\chi =\left( \begin{array}{c} \chi ^{+A} \\ \chi ^{+B} \\ \chi ^0 \\ \end{array} \right) \sim \left( 3~, \frac{\beta }{\sqrt{3}}\right) , \nonumber \\&\rho =\left( \begin{array}{c} \rho ^+ \\ \rho ^0 \\ \rho ^{-B} \\ \end{array} \right) \sim \left( 3~, \frac{1}{2}-\frac{\beta }{2\sqrt{3}}\right) \nonumber \\&\eta =\left( \begin{array}{c} \eta ^0 \\ \eta ^- \\ \eta ^{-A} \\ \end{array} \right) \sim \left( 3~, -\frac{1}{2}-\frac{\beta }{2\sqrt{3}}\right) , \end{aligned}$$\end{document}where A, B denote electric charges as defined in Eq. (7). These Higgses develop vacuum expectation values (VEVs) defined as9⟨χ⟩=1200u,⟨ρ⟩=120v0,⟨η⟩=12v′00.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\langle \chi \rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ 0 \\ u \\ \end{array} \right) , \quad \langle \rho \rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ v \\ 0 \\ \end{array} \right) ,\nonumber \\&\langle \eta \rangle = \frac{1}{\sqrt{2}}\left( \begin{array}{c} v' \\ 0 \\ 0 \\ \end{array} \right) . \end{aligned}$$\end{document}The symmetry breaking happens in two steps: SU(3)L⊗U(1)X→uSU(2)L⊗U(1)Y→v,v′U(1)Q\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L\otimes U(1)_X\xrightarrow {u} SU(2)_L\otimes U(1)_Y\xrightarrow {v,v'} U(1)_Q$$\end{document}. It is therefore reasonable to assume that u>v,v′\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u > v,v'$$\end{document}. After the first step, five gauge bosons will be massive and the remaining four massless gauge bosons can be identified with the before-symmetry-breaking SM gauge bosons. This leads to the following matching condition for the couplings:10g2=g,g1=gXg6g2+β2gX2,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} g_2 = g,\quad g_1 = g_X\frac{g}{\sqrt{6g^2 + \beta ^2 g_X^2}}, \end{aligned}$$\end{document}where g2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_2$$\end{document} and g1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_1$$\end{document} are the two couplings of the SM corresponding to SU(2)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(2)_L$$\end{document} and U(1)Y\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U(1)_Y$$\end{document}, respectively. From this we get the following important equation, which helps to constrain β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}:11gX2g2=6sW21-(1+β2)sW2,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{g_X^2}{g^2} = \frac{6s_W^2}{1-(1+\beta ^2)s_W^2}, \end{aligned}$$\end{document}where the Weinberg mixing angle is defined as tW=tanθW=g1/g2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_W = \tan \theta _W = g_1/g_2$$\end{document} and we denote sW=sinθW\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_W = \sin \theta _W$$\end{document}. Putting in the value of sW\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_W$$\end{document}, we get approximately12|β|≤3,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} |\beta | \le \sqrt{3}, \end{aligned}$$\end{document}which will be used in the numerical analysis.

The masses of the charged gauge bosons are13mY±A2=g24(u2+v′2),mV±B2=g24(u2+v2),mW±2=g24(v2+v′2).\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} m^2_{Y^{\pm A}}= & {} \frac{g^2}{4}(u^2+v^{\prime 2}),\quad m^2_{V^{\pm B}}=\frac{g^2}{4}(u^2+v^2),\nonumber \\ m^2_{W^\pm }= & {} \frac{g^2}{4}(v^2 + v^{\prime 2}). \end{aligned}$$\end{document}We now discuss the mixings of leptons. In general, the mixing between a SM lepton and a new lepton is allowed if they have the same electric charge. However, since we consider a general class of models with arbitrary β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}, this mixing effect will be neglected. This is justified because we will assume that the new leptons are much heavier than the SM leptons. Therefore, only generation mixings as in the SM are allowed. The Yukawa Lagrangian related to these mixings reads14Lleptonyuk=-YabeL′¯aLη∗ebR′-YabνL′¯aLρ∗νbR′-YabEL′¯aLχ∗EbR′+h.c.,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}^\text {yuk}_{\mathrm {lepton}}= & {} -Y^e_{ab} \overline{L'}_{aL} \eta ^*e'_{bR}- Y^\nu _{ab} \overline{L'}_{aL} \rho ^*\nu '_{bR} \nonumber \\&- Y^E_{ab} \overline{L'}_{aL} \chi ^*E'_{bR}+\mathrm {h.c.}, \end{aligned}$$\end{document}where a,b=e,μ,τ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a,b=e,\mu ,\tau $$\end{document} are family indices. The corresponding mass terms are:15Lleptonmass=-Yabev′2e′¯aLebR′+Yabνv2ν′¯aLνbR′-YabEu2E′¯aLEbR′+h.c..\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}^\text {mass}_{\mathrm {lepton}}= & {} -\frac{Y^e_{ab}v'}{\sqrt{2}} \overline{e'}_{aL}e'_{bR}+\frac{Y^\nu _{ab}v}{\sqrt{2}} \overline{\nu '}_{aL} \nu '_{bR}\nonumber \\&- \frac{Y^E_{ab}u}{\sqrt{2}} \overline{E'}_{aL} E'_{bR}+\mathrm {h.c.}. \end{aligned}$$\end{document}From now on we will work in the basis where the SM charged leptons are in their mass eigenstates. This can always be done without loss of generality. We can therefore set Yabe\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y^e_{ab}$$\end{document} to be diagonal and e′=e\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e' = e$$\end{document} in Eqs. (14,15). The transformations from the flavor states to mass eigenstates are defined as16νaL′=UabLνbL,νaR′=UabRνbR,EaL′=VabLEbL,EaR′=VabREbR,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nu '_{aL}= & {} U^L_{ab} \nu _{bL}, \quad \nu '_{aR} = U^R_{ab} \nu _{bR},\nonumber \\ E'_{aL}= & {} V^L_{ab} E_{bL}, \quad E'_{aR} = V^R_{ab} E_{bR}, \end{aligned}$$\end{document}where UL,R\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U^{L,R}$$\end{document} and VL,R\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V^{L,R}$$\end{document} are 3×3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3\times 3$$\end{document} unitary matrices for the neutrinos and new leptons, respectively. The matrix VL\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V^L$$\end{document}, included in the vertices of the SM charged leptons and the new leptons, is similar to the matrix UL=UPMNS\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U^L=U_\text {PMNS}$$\end{document}.

For the Higgs sector, we assume A,B≠0,±1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A,B \ne 0, \pm 1$$\end{document}, so that only the following mixings of scalar fields with the same electric charge are allowed: (χA,ηA)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\chi _{A},~\eta _{A})$$\end{document}, (χB,ρB)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\chi _{B},~\rho _{B})$$\end{document}, (ρ+,η+)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\rho ^+,\eta ^+)$$\end{document} and (χ0,ρ0,η0)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\chi ^0,~\rho ^0,~ \eta ^0)$$\end{document}. The neutral components are expanded as17χ0=12u+ξχ+iζχ,⟨ξχ⟩=⟨ζχ⟩=0,ρ0=12v+ξρ+iζρ,⟨ξρ⟩=⟨ζρ⟩=0,η0=12v′+ξη+iζη,⟨ξη⟩=⟨ζη⟩=0.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\chi ^0=\frac{1}{\sqrt{2}} \left( u+\xi _{\chi }+i \zeta _{\chi }\right) , \quad \langle \xi _{\chi }\rangle =\langle \zeta _{\chi }\rangle =0, \nonumber \\&\rho ^0=\frac{1}{\sqrt{2}} \left( v+\xi _{\rho }+i \zeta _{\rho }\right) , \quad \langle \xi _{\rho }\rangle =\langle \zeta _{\rho }\rangle =0, \nonumber \\&\eta ^0=\frac{1}{\sqrt{2}} \left( v'+\xi _{\eta }+i \zeta _{\eta }\right) , \quad \langle \xi _{\eta }\rangle =\langle \zeta _{\eta }\rangle =0. \end{aligned}$$\end{document}The ratios between VEVs are used to define three mixing angles:18sv′v2=sin2βv′v=v′2v2+v′2,svu2=sin2βvu=v2u2+v2,sv′u2=sin2βv′u=v′2v′2+u2.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} s^2_{v'v}= & {} \sin ^2\beta _{v'v}= \frac{v'^2}{v^2+v'^2}, \; s^2_{vu}=\sin ^2\beta _{vu}= \frac{v^2}{u^2+v^2}, \;\nonumber \\ s^2_{v'u}= & {} \sin ^2\beta _{v'u}= \frac{v'^2}{v'^2+u^2}. \end{aligned}$$\end{document}We will also use the following notation: tv′v=sv′v/cv′v\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{v'v}=s_{v'v}/c_{v'v}$$\end{document}, tv′u=sv′u/cv′u\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{v'u}=s_{v'u}/c_{v'u}$$\end{document}.

The scalar potential is19Vh=μ12η†η+μ22ρ†ρ+μ32χ†χ+λ1η†η2+λ2ρ†ρ2+λ3χ†χ2+λ12(η†η)(ρ†ρ)+λ13(η†η)(χ†χ)+λ23(ρ†ρ)(χ†χ)+λ~12(η†ρ)(ρ†η)+λ~13(η†χ)(χ†η)+λ~23(ρ†χ)(χ†ρ)+2fϵijkηiρjχk+h.c..\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} V_{\mathrm {h}}= & {} \mu _1^2 \eta ^{\dagger }\eta +\mu _2^2\rho ^{\dagger }\rho +\mu _3^2\chi ^{\dagger }\chi +\lambda _1 \left( \eta ^{\dagger }\eta \right) ^2 +\lambda _2\left( \rho ^{\dagger }\rho \right) ^2 \nonumber \\&+\lambda _3\left( \chi ^{\dagger }\chi \right) ^2\nonumber \\&+ \lambda _{12}(\eta ^{\dagger }\eta )(\rho ^{\dagger }\rho ) +\lambda _{13}(\eta ^{\dagger }\eta )(\chi ^{\dagger }\chi ) +\lambda _{23}(\rho ^{\dagger }\rho )(\chi ^{\dagger }\chi )\nonumber \\&+\tilde{\lambda }_{12} (\eta ^{\dagger }\rho )(\rho ^{\dagger }\eta ) +\tilde{\lambda }_{13} (\eta ^{\dagger }\chi )(\chi ^{\dagger }\eta ) +\tilde{\lambda }_{23} (\rho ^{\dagger }\chi )(\chi ^{\dagger }\rho )\nonumber \\&+\sqrt{2} f\left( \epsilon _{ijk}\eta ^i\rho ^j\chi ^k +\mathrm {h.c.} \right) . \end{aligned}$$\end{document}With the above notation, the mass eigenstates are20ϕW±H±=cv′v-sv′vsv′vcv′vρ±η±,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \begin{array}{c} \phi ^{\pm }_W \\ H^{\pm } \\ \end{array} \right)= & {} \left( \begin{array}{cc} c_{v'v} &{} -s_{v'v} \\ s_{v'v} &{} c_{v'v} \\ \end{array} \right) \left( \begin{array}{c} \rho ^{\pm } \\ \eta ^{\pm } \\ \end{array} \right) , \end{aligned}$$\end{document}21ϕY±AH±A=sv′u-cv′ucv′usv′uη±Aχ±A,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \begin{array}{c} \phi ^{ \pm A}_Y \\ H^{\pm A} \\ \end{array} \right)= & {} \left( \begin{array}{cc} s_{v'u} &{} -c_{v'u} \\ c_{v'u} &{} s_{v'u} \\ \end{array} \right) \left( \begin{array}{c} \eta ^{\pm A} \\ \chi ^{\pm A} \\ \end{array} \right) , \end{aligned}$$\end{document}22ϕV±BH±B=svu-cvucvusvuρ±Bχ±B,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \begin{array}{c} \phi ^{\pm B}_V \\ H^{\pm B} \\ \end{array} \right)= & {} \left( \begin{array}{cc} s_{vu} &{} -c_{vu} \\ c_{vu} &{} s_{vu} \\ \end{array} \right) \left( \begin{array}{c} \rho ^{\pm B} \\ \chi ^{\pm B} \\ \end{array} \right) , \end{aligned}$$\end{document}where ϕW±\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi _W^\pm $$\end{document}, ϕY±A\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{ \pm A}_Y$$\end{document} and ϕV±B\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{\pm B}_V$$\end{document} are the Goldstone bosons of W±\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^\pm $$\end{document}, Y±A\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y^{\pm A}$$\end{document} and V±B\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V^{\pm B}$$\end{document}, respectively. The masses of the charged Higgs bosons are23mH±2=(v2+v′2)-fuv′v+12λ~12,mH±A2=(u2+v′2)-fvv′u+12λ~13,mH±B2=(u2+v2)-fv′uv+12λ~23.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&m^2_{H^{\pm }}=(v^2 + v^{\prime 2})\left( \frac{-fu}{v'v}+\frac{1}{2}\tilde{\lambda }_{12}\right) ,\nonumber \\&m^2_{H^{\pm A}}=(u^2+v^{\prime 2})\left( \frac{-fv}{v'u}+\frac{1}{2}\tilde{\lambda }_{13}\right) ,\nonumber \\&m^2_{H^{\pm B}}=(u^2+v^2)\left( \frac{-fv'}{uv}+\frac{1}{2}\tilde{\lambda }_{23}\right) . \end{aligned}$$\end{document}The neutral Higgs bosons are not involved in our calculation; hence they have been ignored. In total, there are six charged Higgs bosons, one neutral pseudoscalar Higgs and three neutral scalar Higgses. Bosonic particles with electric charges of ±B\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm B$$\end{document} are not involved in the present calculation. Nevertheless, their masses and mixing angles are provided above for the sake of completeness.

From the above information we can obtain all vertices needed for the calculation of li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i \rightarrow l_j \gamma $$\end{document} decays. They are listed in Table 1.Table 1

Vertices and couplings for li→ljγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_i\rightarrow l_j\gamma $$\end{document} decays in the 3-3-1 model with arbitrary β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document} and new leptons. All momenta are defined as incoming. The photon field is denoted as Aμ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A^\mu $$\end{document}, a,b=1,2,3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a,b = 1,2,3$$\end{document} are family indices and Γλμν(p1,p2,p3)=(p1-p2)νgλμ+(p2-p3)λgμν+(p3-p1)μgνλ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Gamma _{\lambda \mu \nu }(p_1,p_2,p_3)=(p_1-p_2)_\nu g_{ \lambda \mu } + (p_2-p_3)_\lambda g_{\mu \nu }+(p_3-p_1)_\mu g_{ \nu \lambda }$$\end{document}. Other notations are defined in the text

Equipped with the above Feynman rules, we can proceed to calculate the partial decay width of l1→l2γ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_1 \rightarrow l_2 \gamma $$\end{document} using standard techniques of one-loop calculation. We have done this in a careful way, with at least two independent calculations, and paid special attention to the relative sign between the gauge and Higgs contributions. This relative sign is very important because, as we will see in the numerical results, the interference term can be positive or negative.

In the literature, the calculation of l1→l2γ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_1 \rightarrow l_2 \gamma $$\end{document} is usually done by neglecting the external lepton masses. As stated in the introduction, we found this uneasy because the neutrino masses, which are much smaller than the lepton masses, are kept. We therefore want to check the validity of this approximation. To achieve this we have to keep the external lepton masses.

We have calculated the partial decay width of l1→l2γ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_1 \rightarrow l_2 \gamma $$\end{document} from scratch without approximation. In the following we summarize the key points and present exact analytical results. Results for the SM case are obtained as a special case and are discussed in Sect. 4.1.

We consider the process24l1(p1)→l2(p2)+γ(q),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} l_1(p_1) \rightarrow l_2(p_2) + \gamma (q), \end{aligned}$$\end{document}where p1=p2+q\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_1 = p_2+q$$\end{document} and the helicity indices have been omitted for simplicity. The amplitude reads25M=ϵλ(q)u¯1(p1)Γλu2(p2),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {M} = \epsilon ^\lambda (q) \bar{u}_1(p_1)\Gamma _\lambda u_2(p_2), \end{aligned}$$\end{document}where ϵλ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon ^\lambda $$\end{document} is the photon’s polarization vector, Γλ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Gamma _\lambda $$\end{document} are 4×4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$4\times 4$$\end{document} matrices depending on the gamma matrices, external momenta and coupling constants. After requiring the general conditions that the spinors obey the Dirac equations, qμϵμ=0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q^\mu \epsilon _\mu =0$$\end{document}, and qλu¯1(p1)Γλu2(p2)=0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q^\lambda \bar{u}_1(p_1)\Gamma _\lambda u_2(p_2) = 0$$\end{document}, we can prove that the amplitude depends on only two form factors as26M=2(p1·ϵ)CLu¯2(p2)PLu1(p1)+CRu¯2(p2)PRu1(p1)-(m1CR+m2CL)u¯2(p2)/ϵPLu1(p1)-(m1CL+m2CR)u¯2(p2)/ϵPRu1(p1),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {M}= & {} 2(p_1\cdot \epsilon ) \left[ C_L\bar{u}_2(p_2)P_Lu_1(p_1)+C_R\bar{u}_2(p_2)P_Ru_1(p_1)\right] \nonumber \\&- (m_1C_R + m_2C_L) \bar{u}_2(p_2)/\!\!\!\epsilon P_L u_1(p_1) \nonumber \\&- (m_1C_L + m_2C_R)\bar{u}_2(p_2)/\!\!\!\epsilon P_R u_1(p_1), \end{aligned}$$\end{document}where CL,R\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{L,R}$$\end{document} are called form factors, PL=(1-γ5)/2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_L = (1-\gamma _5)/2$$\end{document}, PR=(1+γ5)/2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_R = (1+\gamma _5)/2$$\end{document}. The partial decay width is then written as27Γ(l1→l2γ)=(m12-m22)316πm13|CL|2+|CR|2.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Gamma (l_1\rightarrow l_2\gamma )= \frac{(m^2_1-m^2_2)^3}{16\pi m^3_1}\left( |C_L|^2+|C_R|^2\right) . \end{aligned}$$\end{document}This result is well known and has been given in e.g. Ref. [17].

Since we assume that the exotic leptons are much heavier than the SM leptons, the branching fractions of the dominant decays of l1→l2ν¯2ν1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_1 \rightarrow l_2 \bar{\nu }_2 \nu _1$$\end{document} in the 3-3-1 model are the same as those of the SM. Using the well-known tree-level result of Γ(l1→l2ν¯2ν1)=GF2m15/(192π3)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Gamma (l_1 \rightarrow l_2 \bar{\nu }_2 \nu _1) = G_F^2 m_1^5/(192\pi ^3)$$\end{document} (see e.g. [18]), where GF\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_F$$\end{document} is the Fermi coupling constant, we write the branching fraction as28Br(l1→l2γ)=12π2GF2|DL|2+|DR|2Br(l1→l2ν¯2ν1),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Br}(l_1 \rightarrow l_2 \gamma ) = \frac{12\pi ^2}{G_F^2}\left( |D_L|^2+|D_R|^2\right) \text {Br}(l_1 \rightarrow l_2 \bar{\nu }_2 \nu _1),\nonumber \\ \end{aligned}$$\end{document}where GF=g2/(42mW2)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_F=g^2/(4\sqrt{2}m^2_W)$$\end{document} and we have defined CL,R=m1DL,R\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{L,R} = m_1 D_{L,R}$$\end{document} and the approximation m2≪m1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_2 \ll m_1$$\end{document} has been used for the first factor, but not for DL,R\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{L,R}$$\end{document}. For later numerical analysis we will use Br(μ→eν¯eνμ)=100%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Br}(\mu \rightarrow e \bar{\nu }_e \nu _\mu ) = 100\%$$\end{document}, Br(τ→eν¯eντ)=17.82%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Br}(\tau \rightarrow e \bar{\nu }_e \nu _\tau ) = 17.82\%$$\end{document} and Br(τ→μν¯μντ)=17.39%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Br}(\tau \rightarrow \mu \bar{\nu }_\mu \nu _\tau ) = 17.39\%$$\end{document} as given in Ref. [1]. It is noted that DL∝O(m2/m1)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_L\propto \mathcal {O}(m_2/m_1)$$\end{document} (since only a left-handed electron can participate in SU(3)L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SU(3)_L$$\end{document} interactions) and DR∝O(1)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_R\propto \mathcal {O}(1)$$\end{document}, and hence, in the approximation m2≪m1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_2 \ll m_1$$\end{document}, we have Br(l1→l2γ)∝|DR|2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Br}(l_1 \rightarrow l_2 \gamma ) \propto |D_R|^2$$\end{document}. This point is important to understand the approximate results discussed in the next sections.Fig. 1

Representative Feynman diagrams contributing to l1→l2γ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_1\rightarrow l_2 \gamma $$\end{document} decays. There are two groups: the neutrino contribution (a,d) and the exotic-lepton contribution (b,c,e,f)

Input parameters are specified as follows. We use, according to Ref. [1],31α(0)=1/137.035999679,mW=80.385GeV,me=0.5109989461MeV,mμ=105.6583745MeV,mτ=1776.86MeV,Δm212=7.53×10-5eV2,Δm322=2.45×10-3eV2,sin2(θ12)=0.307,sin2(θ13)=0.021,sin2(θ23)=0.51.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\alpha (0) = 1/137.035999679,\quad m_W = 80.385 {\,\text {GeV}},\nonumber \\&m_e = 0.5109989461 {\,\text {MeV}}, \quad m_{\mu } = 105.6583745 {\,\text {MeV}}, \nonumber \\&m_{\tau } = 1776.86 {\,\text {MeV}},\nonumber \\&\Delta m^2_{21}=7.53\times 10^{-5} {\,\text {eV}}^{2}, \quad \Delta m^2_{32}=2.45\times 10^{-3} {\,\text {eV}}^2,\nonumber \\&\sin ^2(\theta _{12})=0.307, \quad \sin ^2(\theta _{13}) = 0.021, \quad \sin ^2(\theta _{23}) = 0.51.\nonumber \\ \end{aligned}$$\end{document}The neutrino mixing matrix is assumed to be real and is calculated from the above mixing angles as32UL=c12c13s12c13s13-s12c23-c12s23s13c12c23-s12s23s13s23c13s12s23-c12c23s13-c12s23-s12c23s13c23c13,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} U^L = \left( \begin{array}{ccc} c_{12}c_{13} &{} s_{12}c_{13} &{} s_{13} \\ -s_{12}c_{23}-c_{12}s_{23}s_{13} &{} c_{12}c_{23} - s_{12}s_{23}s_{13} &{} s_{23}c_{13}\\ s_{12}s_{23} - c_{12}c_{23}s_{13} &{} -c_{12}s_{23} - s_{12}c_{23}s_{13} &{} c_{23}c_{13} \end{array} \right) ,\nonumber \\ \end{aligned}$$\end{document}where cij=cosθij\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_{ij}=\cos \theta _{ij}$$\end{document}, sij=sinθij\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{ij}=\sin \theta _{ij}$$\end{document} with i,j=1,2,3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i,j=1,2,3$$\end{document}.