Image scene classification aiming to assign specific semantic labels for each image is vitally important for the applications of remote sensing (RS) data. In real world, since the observation environment is open and dynamic, RS images are collected sequentially and the numbers of images and classes grow rapidly over time. Most existing scene classification methods are offline learning algorithms, which are inefficient and unscalable for this scenario. In this article, an incremental learning with open-set recognition (ILOSR) framework is proposed for RS image scene classification in the open and dynamic environment, which can identify the unknown classes from a stream of data and learn these new classes incrementally. Specifically, a controllable convex hull-based exemplar selection strategy is designed to address the catastrophic forgetting issue in incremental learning, which can reduce training time and memory footprint effectively. In addition, a new loss function based on prototype learning and uncertainty measurement is proposed for OSR to enhance the interclass discrimination and intraclass compactness of the learned deep features. Experimental results on real RS datasets demonstrate that the proposed method can not only outperform the state-of-the-art approaches on offline classification, incremental learning, and OSR problem separately but also achieve better and more stable performance in the experiments for ILOSR.