## Hard Mathematical Problems in Cryptography and Coding Theory

##### Abstract

In this thesis, we are concerned with certain interesting computationally hard problems and the complexities of their associated algorithms. All of these problems share a common feature in that they all arise from, or have applications to, cryptography, or the theory of error correcting codes. Each chapter in the thesis is based on a stand-alone paper which attacks a particular hard problem. The problems and the techniques employed in attacking them are described in detail. The first problem concerns integer factorization: given a positive integer \(N\). the problem is to find the unique prime factors of \(N\). This problem, which was historically of only academic interest to number theorists, has in recent decades assumed a central importance in public-key cryptography. We propose a method for factorizing a given integer using a graph-theoretic algorithm employing Binary Decision Diagrams (BDD). The second problem that we consider is related to the classification of certain naturally arising classes of error correcting codes, called self-dual additive codes over the finite field of four elements, \(GF(4)\). We address the problem of classifying self-dual additive codes, determining their weight enumerators, and computing their minimum distance. There is a natural relation between self-dual additive codes over \(GF(4)\) and graphs via isotropic systems. Utilizing the properties of the corresponding graphs, and again employing Binary Decision Diagrams (BDD) to compute the weight enumerators, we can obtain a theoretical speed up of the previously developed algorithm for the classification of these codes. The third problem that we investigate deals with one of the central issues in cryptography, which has historical origins in the theory of geometry of numbers, namely the shortest vector problem in lattices. One method which is used both in theory and practice to solve the shortest vector problem is by enumeration algorithms. Lattice enumeration is an exhaustive search whose goal is to find the shortest vector given a lattice basis as input. In our work, we focus on speeding up the lattice enumeration algorithm, and we propose two new ideas to this end. The shortest vector in a lattice can be written as \({\bf s} = v_1{\bf b}_1+v_2{\bf b}_2+\ldots+v_n{\bf b}_n\). where \(v_i \in \mathbb{Z}\) are integer coefficients and \({\bf b}_i\) are the lattice basis vectors. We propose an enumeration algorithm, called hybrid enumeration, which is a greedy approach for computing a short interval of possible integer values for the coefficients \(v_i\) of a shortest lattice vector. Second, we provide an algorithm for estimating the signs \(+\) or \(-\) of the coefficients \(v_1,v_2,\ldots,v_n\) of a shortest vector \({\bf s}=\sum_{i=1}^{n} v_i{\bf b}_i\). Both of these algorithms results in a reduction in the number of nodes in the search tree. Finally, the fourth problem that we deal with arises in the arithmetic of the class groups of imaginary quadratic fields. We follow the results of Soleng and Gillibert pertaining to the class numbers of some sequence of imaginary quadratic fields arising in the arithmetic of elliptic and hyperelliptic curves and compute a bound on the effective estimates for the orders of class groups of a family of imaginary quadratic number fields. That is, suppose \(f(n)\) is a sequence of positive numbers tending to infinity. Given any positive real number \(L\). an effective estimate is to find the smallest positive integer \(N = N(L)\) depending on \(L\) such that \(f(n) > L\) for all \(n > N\). In other words, given a constant \(M > 0\). we find a value \(N\) such that the order of the ideal class \(I_n\) in the ring \(R_n\) (provided by the homomorphism in Soleng's paper) is greater than \(M\) for any \(n>N\). In summary, in this thesis we attack some hard problems in computer science arising from arithmetic, geometry of numbers, and coding theory, which have applications in the mathematical foundations of cryptography and error correcting codes.