In a word, yes, it’s okay to drive with the air-conditioner off. Your car’s A/C compressor is an accessory, merely an add-on to your car’s engine. Whether it’s on or off, your car's engine will run fine. Occasionally not using it won’t harm it. Just like not using your car occasionally won’t cause damage. Operating the car without the A/C won’t affect anything other than your own comfort.
But in a way, the assumption is correct. If you hardly ever use your car’s air-conditioner it can have an adverse effect on the system. Because of the inactivity, the oil and the refrigerant inside the compressor aren't circulating and the seals inside the compressor and in the lines may dry up. When the seals dry up it can lead to refrigerant leaks which will cause your A/C to cease blowing cold air.
Hence, it’s a good idea to run it at full blast for at least 10 minutes every other week. That’s the advice that people in colder climates are given. Even though it’s winter time and they don’t need to use the A/C in their car, they are still advised to run it just to circulate the oil in the compressor and the refrigerant in the lines. Switching off a car air-conditioner for weeks on end would be a mistake.
There are also other reasons to run your air-conditioner other than to keep the seals in the compressor and the lines in good condition. Prolonged non-usage will promote moisture build-up in the ducts that pipes cold air to the vents. Because of this moisture, bacteria and fungi could form and subsequently get blown into the car’s cabin when you start the system again after a long period.
Using your air-conditioner also removes moisture from the air inside your car’s cabin, which is very useful during the rainy months. The removal of moisture from the air keeps your windshield and side windows clear of fog and steam. So if your windows are fogging up during the rainy months, turn on your A/C so that you can see out your car. This is a big help as it is no fun to drive while wiping your windows clear of fog and mist.
