Probably many slave holders in the Old South (and North) rationalized what they were doing by saying they were forced to compete with the others in business who were also using slaves, similar to the use of illegal aliens in business today, a big difference a sizable portion of illegal aliens in blue states vote.
