See Section 228.41 of Reg BB below - (d) on limiting, or adjusting the assessment area.
There is always a danger if you start adjusting the area down from an entire MSA or an entire county. Examiners will immediately wonder if you are limiting in order to try and discriminate. At a previous bank I did just what you are proposing - we were expanding and had put one branch location in a county, logically we could not expect to serve the entire county from the one location BUT examiners did not like the limitation. So my experience says your logic for excluding certain census tracts or parts of a county might pass muster only if what you are excluding is high income areas ???
(c) Geographic area(s) for other banks. The assessment area(s) for a bank other than a wholesale or limited purpose bank must:
(1) Consist generally of one or more MSAs or metropolitan divisions (using the MSA or metropolitan division boundaries that were in effect as of January 1 of the calendar year in which the delineation is made) or one or more contiguous political subdivisions, such as counties, cities, or towns; and
(2) Include the geographies in which the bank has its main office, its branches, and its deposit-taking ATMs, as well as the surrounding geographies in which the bank has originated or purchased a substantial portion of its loans (including home mortgage loans, small business and small farm loans, and any other loans the bank chooses, such as those consumer loans on which the bank elects to have its performance assessed).
(d) Adjustments to geographic area(s). A bank may adjust the boundaries of its assessment area(s) to include only the portion of a political subdivision that it reasonably can be expected to serve. An adjustment is particularly appropriate in the case of an assessment area that otherwise would be extremely large, of unusual configuration, or divided by significant geographic barriers.
_________________________
My opinions are just that, and might be worth what you paid for them.