“holding in moisture” is not the same as having damp hair. Holding in moisture is more of a microscopic concept related to your hair health. Keeping hair hydrated is important to preventing dry hair, split ends, and other problems.
When you seal moisture into your hair, this does not encourage mildew or buildup. You will need to wash the oil out of your hair, but it never should be applied so thickly that it causes buildup in the first place. Just wash your hair as normal (which should be thoroughly, anyway), and everything should be fine.
Please be aware this recommendation is mostly for dry hair types, and silky/oily hair types can add oil (sparingly) in whatever way they choose. Judging by the fact that you wash your hair every three days, which is somewhat frequent, I don’t think ensuring your hair is damp before dabbing oil onto it is at all necessary.
Let me know if I misinterpreted your question or statements!