In this paper, decentralized games of discrete-time large population stochastic multi-agent systems are considered under a coupled quadratic performance index. Based on the state aggregation method, the estimate of the population state average is constructed, with which and the Nash certainty equivalence principle, the decentralized control law is designed. By the probability limit theory, the stability and optimality of closed-loop system is analyzed. The main results are: (1) The estimate of the PSA is shown to be strongly consistent in some norm sense, which implies that the estimation error is convergent to zero almost surely as the number of agents increases to infinity. (2) The closed-loop system is almost surely uniformly stable, in other words, the stability is independent of the number of system agents. (3) The decentralized control law is an almost surely asymptotic Nash equilibrium strategy.