Security deposit Returnable?

My fianc� and I looked at an apt and liked it. We gave the guy the sec deposit and let him know we couldn't move till the 15th even tho he had it ready for the first. My fianc� got a letter for vt state trooper and part of the requiremets are to establish residency in vt... We live in ny. We called the landlord back today, 4 days after we gave the deposit to say we cannot move. It's before the first of the month and he won't give it back to us unless he rents it. We didn't sign anything so can he legally do this?