Notice: Historically, sneak has been one of the features of the game that we're broken as a side effect of other changes many times. Because of that, how exactly is sneak stuff effective all depends on the patch. There have been ones where sneak stuff gave nothing, there were ones where achieving maximum sneak had no requirements (just crouch), we've seen a lot of contradicting results :D
It's not necessary to be invisible to be unVATSable - its necessary to have enough sneak to put against opponents perception.
In practice, there are three ranges. Hard VATS, easy VATS, VATS enabled. Quite often, the difference between the first two is hard to notice, as for given perception given sneaker might put them so close to each other one of them practically doesn't exist.
VATS enabled area is the area within 10 meters from you. All sneaking, visible targets are VATSable when in this range
Easy VATS is a further range. To start VATSing an opponent that's sneaking and that's visible, you need to hit them manually. It doesn't mean what will you hit with, as long as you deal damage, so high AoE grenades work quite nicely at flushing people out of hiding and applying the VATS mark. Easy VATS is the range in which given sneaker is quite easy to stay targetable after being spotted, before you lose ability to track them.
Hard VATS is a range in which given sneaker, after being hit, is unable to be VATSed extremely quickly, practically without giving you the chance to stack Concentrated Fire.
So, basically, if you want to VATS a target that's sneaking, hit them once will enable it for a quite while, until they become invisible or their sneak kicks back in.
To kick a chameleon armor wielding opponent out of their invisibility, you must stagger them so that they move out of their position.
The maximum range in which targets can be VATSed is 50 meters.