I gave "Teach Your Own" an honest try and made it through about half before I speed read the remainder. Holt was an educator, himself, and later a researcher into the learning of young children. I classify his educational view as one which is in support of a more Montessori type approach to teaching. Without putting words of my own into his mouth, his viewpoint stems from the premise that babies come out of the womb interested and excited to learn and that adults often get in the way. (Must be mentioned... I couldn't find a photo for the original "Teach Your Own", and I might, yet, give this one pictured above specifically for homeschooling a try.)
Overall, I found Holt's writing, while informative, to lack inspiration in the day to day of homeschooling. I recommend reading one of his books (over a break, maybe, when more time allows) just for frame of reference in other writings or discussions which might reference this well known child researcher.
(As a side, Holt's writing does come across as negative toward public schooling and formal schooling in general. Though maybe he presents his theories in such a manner as a cautionary tale of "over teaching" and not necessarily anti-formal education...but that's for you to read and determine!)
Discuss amongst yourselves!