In super duper dumb terms, CAS latency numbers indicate the time it takes to do something, so:
5 units of time is more than 4 units of time so 4 is faster (less time) than 5.
But that's hugely simplified because there are a bunch of factors involved.
As Flying Fox said, CAS timings are not worth messing with unless your some Liq-Nitro freak hell bent on beating someone else's benchmark score even if by only 1/100th of a percent. Sometimes those timings can have 1-3% impact and in rare instances the impact can be higher but those are rare and very specific instances.