You can't break up a census tract as KC has said. It is the smallest geographic unit allowed in CRA. But remember, the examiners won't be able to breakdown your lending below that level too. So if you don't lend in those other towns in the tract it won't affect your performance evaluation as long as you are generating sufficient lending in the entire tract, even if all of it is in the town inside your AA and not in the other towns.
_________________________
CRA Exam Preparation, CRA Performance Evaluations, Key Performance Benchmarks, & maps