What do you do when you can't find what you're looking for? Ask! The people who work with collections every day are going to be the experts on what's in their holdings. So never hesitate to reach out to a library or archive's reference department. They are there to help!
Most rare book repositories will contain both library holdings and archival holdings. Every repository organizes their materials differently for a number of reasons: staffing, size of collections, breadth of collections, etc. Books, however, will always be listed in what's called a Catalog.
Most catalogs started out as giant card filing cabinets with 1 or more cards per book (depending on if all of the information fit on 1 card). Public libraries are quickly moving from physical catalogs to online catalogs, but rare book libraries are a little more behind. Most places will have an online catalog, but it is not guaranteed that all of the catalog entries from the physical catalog will have been converted into online entries.
All of this to say, when you are searching for a printed book, you need to search a library's catalog (or reach out to their reference department) to know what they have in their holdings.
Manuscripts, on the other hand, will be found in an archive, which usually has a separate online database you'll have to search in order to find those materials. Manuscript collections will also have what is called a Finding Aid. These are documents created by the archivist who process the collection that tell you what is in a collection. The level of detail on finding aids will differ from archive to archive and collection to collection, so if you encounter a finding aid that is just bare bones, reach out to the archive's reference department and see if they can give you a better idea of the contents of a collection.
Manuscript collections can range from one box with a few pieces of paper in it to literal hundreds of boxes with thousands of pages of materials. What this means for researchers, is there might be a LOT of stuff you have to sift through to find what you need. What this means for library workers is a lot of time spent on one project. Most institutions only have one archivist who is in charge of processing new collections and getting them into the library/archive's database. This is a task that requires a lot of time and attention to detail. All of this to say, please be kind to and patient with library workers as they do their best to fulfil your requests.
When institutions acquire new materials, they have to be processed into the collections. For books, this means being catalogued by a cataloguing librarian and for archival materials this means being processed by an archivist.
When a book is catalogued, it is assigned a call number, usually made from the Library of Congress classification system. Each volume will have its own assigned number that will look something like this: PR4611.A55 D33.
For archives, rather than each item having a call number, each collection has a call number. So every time in an archival collection will be filed under the same number, and the way that you find specific items in a collection is either by looking through the Finding Aid or asking the archival staff.