A brain tumor is an abnormal mass or growth of a cell that leads to certain death, and this is still a challenging task in clinical practice. Early and correct diagnosis of this type of cancer is very important for the treatment process. For this reason, this study aimed to develop computer-aided systems for the diagnosis of brain tumors. In this research, we proposed three different end-to-end deep learning approaches for analyzing effects of local and deep features for brain MRI (...) images anomaly detection. The first proposed system is Directional Bit-Planes Deep Autoencoder which extracts and learns local and direction features. The DBP-DAE by decomposition of a local binary pattern into eight bit-planes extracts are directional and inherent local-structure features from the input image and learns robust feature for classification purposes. The second one is a Dilated Separable Residual Convolutional Network which extracts high and low-level features. The main advantage of this approach is that it is robust and shows stable results regardless to size of image database and to solve overfitting problems. To explore the effects of mixture of local and deep extracted feature on accuracy of classification of brain anomaly, a multibranch convolutional neural network approach is proposed. This approach is designed according to combination of DBP-DAE and DSRCN in an end-to-end manner. Extensive experiments conducted based on brain tumor in MRI image public access databases and achieves significant results compared to state-of-the-art algorithms. In addition, we discussed the effectiveness and applicability of CNNs with a variety of different features and architectures for brain abnormalities such as Alzheimer’s. (shrink)