The standard model (SM) was well established by the discovery of the Higgs boson. The SM, however, does not answer a fundamental question about the origin of flavor structure. In order to understand this, many works have addressed the discrete groups for flavors. The group was used in early models of quark masses and mixing angles [1,2]. This group was also studied to explain the large mixing angle [3] in the oscillation of atmospheric neutrinos [4]. After the discovery of the neutrino oscillations, the discrete symmetries of flavors have been developed to reproduce the observed lepton mixing angles [5–13].

Superstring theory with certain compactifications can lead to non-Abelian discrete flavor symmetries. (See, e.g., Refs. [14–20].) The torus and orbifold compactifications have the modular symmetry of the modulus parameter. The flavors of both quarks and leptons transform non-trivially under the modular transformation [21–27]. In this sense, the modular symmetry is a non-Abelian discrete flavor symmetry. Yukawa and other couplings depend on the moduli parameters in four-dimensional low-energy effective field theory derived from superstring theory. Each coupling therefore transforms non-trivially under the modular symmetry, which is an important difference from the conventional flavor symmetries.

The modular group includes , , , and as its finite subgroups [28]. An attractive flavor model has been put forward based on the modular group [29]. This work stimulates model building based on [30], [31], and [32]. Phenomenological discussions of neutrino flavor mixing have been presented based on the [33,34], [35], and [36] modular groups. In particular, comprehensive analysis of the modular group has provided a distinct prediction of the neutrino mixing angles and the charge–parity (CP) violating phase [34]. Applications of the modular symmetry have begun to develop in quark and lepton flavors. The modular symmetry has also been applied to the SU(5) grand unified theory (GUT) of quarks and leptons [37], while the residual symmetry of the modular symmetry has been investigated phenomenologically [38]. The modular forms for and have also been constructed [39], and the extension of the traditional flavor group has been discussed with modular symmetries [40]. Moreover, multiple modular symmetries are proposed as the origin of flavor [41]. The modular invariance has also been studied combined with generalized CP symmetries for theories of flavors [42]. The quark mass matrix has been discussed in the and modular symmetries as well [43,44]. Besides the mass matrices of quarks and leptons, related topics such as baryon number violation [43], dark matter [45], radiatively induced neutrino masses [46], and the modular symmetry anomaly [47] have been discussed.

Among these, the unification of quark and lepton flavors based on the modular symmetry is an important work from the standpoint of quark–lepton unification [37,48] since the modulus is common to both quarks and leptons. In this paper we construct an flavor model with modular invariance in the framework of SU(5) GUT and discuss the Dirac CP violating phases in both quark and lepton sectors as well as the neutrino masses and mixing, the effective neutrino mass of the neutrinoless double beta decay, and Majorana CP violating phases. We consider a six-dimensional compact space in addition to our four-dimensional spacetime in superstring theory. Suppose that the six-dimensional compact space has some constituent spaces and that they include a two-dimensional compact space . Note that can have geometrical symmetry such as the modular symmetry. Quark mixing and lepton mixing are explained by a single flavor symmetry originating from . The modular forms for the quark and lepton sectors are the same and determined by a common value of in our setup. The other four-dimensional part of may contribute to an overall factor of the Yukawa couplings, but not to their ratios.

We assume the modular symmetry for flavors of quarks and leptons since it is the minimal non-Abelian discrete symmetry. Furthermore, we assume SU(5) GUT as a first step to building a realistic flavor model with modular invariance for both quarks and leptons. It is emphasized that the vacuum expectation value (VEV) of the 24-dimensional adjoint Higgs multiplet creates a difference between the mass eigenvalues of down-type quarks and charged leptons. Our mass matrices reproduce the observed Cabibbo–Kobayashi–Maskawa (CKM) and Pontecorvo–Maki–Nakagawa–Sakata (PMNS) parameters successfully. We predict the leptonic CP violation phase and the effective mass of the neutrinoless double beta decay versus the sum of neutrino masses, respectively.

This paper is organized as follows. In Sect. 2 we present our SU(5) GUT model with the finite modular symmetry . In Sect. 3, we present numerical analyses of our model. Section 4 is devoted to a summary. Appendix A shows the modular forms of briefly, and Appendix B presents relevant parameters in the lepton flavor mixing.

Let us present our framework in the supersymmetric (SUSY) SU(5) GUT. Matter fields can be accommodated in the and representations as (1) where the subscripts denote the quark colors, the superscript denotes CP-conjugated fermions, and the flavor indices are omitted. In addition, we introduce the right-handed neutrinos , which are SU(5) singlets. We present the charge assignments of superfields for the gauge group, flavor symmetry, and modular weights in Table 1, where the subscript of and denotes the th family. An adjoint representation of scalars breaks the SU(5) gauge symmetry and leads to the mass differences among quarks and charged leptons. The electroweak breaking of the SM is realized by a of Higgs, , which also contribute to the fermion mass matrices. These Higgs multiplets are listed in Table 1, which also presents the modular forms of weights 2 and 4 that we use.

For Yukawa interactions, the modular invariant superpotential is written as (2) where the three terms of the right-hand side lead to the mass terms of up-type quarks, down-type quarks, and charged leptons and neutrinos, respectively. The up-type quark mass matrix is derived from , which is explicitly given as: (3) where , , , and are dimensionless complex constants. Here, denotes the cut-off scale around the SU(5) energy scale. We set . Thus, the next-order corrections of are . We neglect their effect because the experimental values of masses and mixing angles for the quarks and leptons include errors of . We focus on the parameter regions in the following numerical analysis. By using the tensor product of doublets in Appendix A, the mass matrix of up-type quarks is given in terms of the modular forms and of Appendix A as (4) where the argument of the modular forms is omitted, and the parameters are redefined as follows: (5) where is the VEV for the doublet component of . This mass matrix was investigated in our previous work [43].

Suppose the neutrinos to be Majorana particles, which are realized by the seesaw mechanism. Then, the neutrino mass matrix is derived from the superpotential : (6) where –, are dimensionless complex constants. The additional term is the contribution to the right-handed Majorana mass terms from the dimension-five operators, (7) where the are arbitrary coefficients. Here, we take the diagonal basis of .

After integrating out – fields, the Majorana left-handed neutrino mass matrix is therefore given as follows: (8) where the parameters , , , , and are redefined as (9) and (10)

The superpotentials for the down-type quarks and charged leptons are written as (11) where , , , and are dimensionless complex constants. We focus on the parameter region in the following numerical analysis. We can construct a mass matrix for the down-type quarks and charged leptons: (12) where we have introduced a new parameter defined as (13) and is the VEV of the doublet component of . We can obtain a successful mass matrix for the down-type quarks: (14) where we have redefined some parameters as in Eq. (5).

The quark mass matrices in Eqs. (4) and (14) can reproduce the observed CKM mixing matrix elements and quark mass ratios at the GUT scale [49,50]. Indeed, we have obtained successful up-type and down-type quark mass matrices with hierarchical flavor structure which are completely consistent with the observed masses and CKM parameters [43].

Let us discuss the charged lepton mass matrix, which is possibly related to the down-type quark mass matrix, by using the SU(5) GUT relation. We rewrite the coefficients in the down-type quark mass matrix elements in terms of the sum of contributions from VEVs of and as follows: (15) where we have the following relations for the parameters of Eq. (12): (16)

Let us give the Clebsch–Gordan (CG) factor , which is derived by the ratio of VEVs for the charged lepton sector and down-type quark sector: (17) since takes the VEV as . The charged lepton mass matrix is therefore obtained in terms of the elements of the down-type quark mass matrix and the coefficient by transposing the down-type quark mass matrix: (18)

In the quark and lepton sectors, we obtained enough parameter sets including the value of the modulus which reproduce the quark and lepton masses and CKM mixing. For example, we set (19) which lead to and . We show a typical sample of our parameter sets: (20) in GeV units. These are obtained from the parameters (21) in of Eq. (3), and the parameters (22) in of Eq. (11).

This sample parameter set leads to the following result for the CKM mixing parameters: (23) as well as the proper hierarchy of quark and charged lepton masses. We use the above parameters for the prediction of the neutrino sector in the next section.

We have obtained parameter regions that reproduce the observed fermion mass ratios and CKM mixing parameters. Our results are consistent with the experimental results for quark mass ratios and charged lepton mass ratios at the GUT scale within the range [49,50].¹ In the following subsections we present predictions in the neutrino sector and discuss the correlation between the CKM and PMNS mixing parameters.

Since we have separated the parameters of the down-type quarks such as , , , , and into two terms as defined in Eq. (15), we can scan the parameters in the charged lepton mass matrix of Eq. (18) by using the successful parameter sets of the down-type quark sector. A typical sample is presented in Eq. (20). The parameters of the neutrino mass matrix of Eq. (8) have been scanned in the region of , , and in units, while phases have been scanned in . We present a sample point satisfying the recent neutrino oscillation experimental data [51,52] as well as the fermion mass ratio and CKM mixing parameters at the GUT scale.²

We have presented a flavor model with the modular invariance in the framework of SU(5) GUT and discussed the CKM and PMNS mixing parameters of both quark and lepton sectors. We have considered a six-dimensional compact space in addition to our four-dimensional space-time and supposed that the six-dimensional compact space has some constituent parts that include a two-dimensional compact space . Then, the quarks and leptons have the same modular symmetry and the same value of in our setup. We note that our model does not require any gauge singlet scalars such as flavons. The difference between the mass eigenvalues of down-type quarks and charged leptons is realized by the 24-dimensional adjoint Higgs multiplet .

The setup of our model is reasonably accepted in the neutrino phenomenology if the right-handed neutrino masses are taken to be around GeV. Their favored ranges are fairly limited; masses larger than GeV are basically disfavored by the perturbativity of the neutrino Yukawa couplings, whereas lower masses require a more significant suppression in the dimension-five operators, or more severe cancellation between the operators and the bare mass terms.

We have analyzed our model numerically and found parameter regions which are consistent with both the observed CKM and PMNS mixing parameters for both NH and IH cases. The predicted Dirac CP violating phase is allowed in the whole range for the NH case. The sum of neutrino masses is larger than meV. For IH, it is predicted in the region of . In particular, it is around near the lower bound of our prediction of the sum of neutrino masses, meV. It is expected to test our model by astronomical observation for the neutrino mass constraint, as well as precise observation for the Dirac CP violating phase.

We have also predicted the effective mass in the neutrinoless double beta decay , which is [meV] for NH and [meV] for IH. The development of experiments for the neutrinoless double beta decay is also expected to test our model. It is also noted that the proton lifetime is sufficiently long compared with the observed lower bound of yrs.

Since our model has a large number of free parameters, distinct correlations between the CKM and PMNS mixing parameters are not found. However, the common value of the modulus is clearly obtained by imposing the experimental data for the CKM and PMNS mixing parameters as well as the quark and lepton masses. If we can build a flavor model with a small number of free parameters in a specific GUT framework, it is expected to find correlations between the CKM and PMNS matrices.