With the growth in applications of Artificial Intelligence day by day, every domain is going automated. Machine learning has enabled systems to learn the process on its own in order to reduce the human labour. In sports like cricket, Football AI has not been used on a greater scale but there are certain areas where it can be of great help to apply AI techniques. In this paper the outcome classification task has been performed on cricket videos. The main purpose of performing such activities is to create automatic commentary generation. There are many sub-tasks needed to be considered for this task. One of those tasks is to classify the outcome of each ball for which commentary is to be generated. There has not been any standard data to perform such task, neither are any benchmark results to compare a new one. So in this paper, from data collection to performing the classification operation with results has been produced. There are four most general outcomes in the game of cricket such as Run, Dot, Boundary, Wicket. With the help of Convolutional Neural Network and Long Short-Term Memory Networks the outcome of cricket match ball by ball videos has been classified with 70% of test accuracy.