In some people the lactase-producing cells in the small intestine will not "atrophy" or diminish if they are not regularly stimulated by the presence of lactose. First of all, your body either makes the lactase enzyme for life or it gradually diminishes production with age. Nothing you eat or don't eat really affects this in any way. (However disease, drugs or damage to the small intestine can also stop production, either temporarily or permanently, but that's a relatively tiny number of cases).
Food intolerances never change. They are based on a lacking of a particular enzyme or catalyst. I find that people can start to tell that they have an intolerance once they do cut out the food completely.You might have some symptoms, though not necessarily G.I. related, that you don't identify until you cut out the source of the problem.
Also cheeses made from milk, goat or cow,will have a reduction in the amount of lactose due to the fermentation process. Cheese is relatively low in lactose in that way and generally speaking, the older the cheese, the less lactose it has. That is why you can tolerate cheese so well.