Sometimes CUDA program crashed during execution, before memory was flushed. As a result, device memory remained occupied.

There are some solutions:

Try using：

Although it should be unecessary to do this in anything other than exceptional circumstances, the recommended way to do this on linux hosts is to unload the nvidia driver by doing

with suitable root privileges and then reloading it with

If the machine is running X11, you will need to stop this manually beforehand, and restart it afterwards. The driver intialisation processes should eliminate any prior state on the device.

This answer has been assembled from comments and posted as a community wiki to get this question off the unanswered list for the CUDA tag

This methods working for me:

check what is using your GPU memory with

Your output will look something like this:

Then kill the PID that you no longer need on htop or with

Or simply reboot:

欢迎转载，转载请保留页面地址。帮助到你的请点个推荐。