I very much like the writing style of Seth Godin, The Dip: A Little Book That Teaches You When to Quit (and When to Stick) excels in plain-English commentary about when to quit and when to stick with it. Seth explains that there are three potential paths to most decisions: The Dip, The "Cul-de-sac" and the Cliff.
The Dip: Your project/initiative/whatever is all fun in the beginning, your all jazzed up and highly motivated. Then, quit all of a sudden somethign happens and things get really hard. Now you have a choice to make, stick with it or cut your losses.
The "Cul-de-sac": French for dead-end, no matter what you do, you spin your wheels and stay stuck in the mud, these are your typical dead-end jobs and go-nowhere projects. Cut losses here!
The clif: Situations where you cannot quit until the whole thing falls apart. Quitting is often difficult but necessary.
Find a way to quit Clifs and "Cul-de-sac" projects, jobs, etc.
Questions to ask yourself before quitting:
Am I panicking?
Who am I trying to influence?
Am I making measurable progress?
So at the end of the day if there is no pay off, quit!