The phrase 'wrong side of the tracks' refers to the direction wind blew dirty train smoke.
From “Why the "wrong side of the tracks" is usually the east side of cities | Stephen DeBerry” — TED
see it said at 2:27
Did you know?
Surprising facts — each proven at the exact second of its video