how do you completely remove a library from the aspect system?

We are doing some clean up work and have a few project specific created libraries that are not being used (confirmed by checking library usage). I know I can remove from Control Builder M but I also want to remove completely from the aspect system. Do I delete from Object type structure or Library structure in plant explorer? I have also created a backup export file of the libraries just in case we ever decide we need them again (highly unlikely).