This paper explores aspects of household livelihoods and welfare among poor rural farmers in Kenya, Uganda and Tanzania. The models use data from an extensive cross-sectional household survey undertaken in 2013 of over 3,500 rural households. Interest is in identifying the main input constraints to improving farm production, with a particular focus on labour constraints, and the impacts of farm production on the well-being of children—their food security, resilience and engagement in school. The results show low agricultural productivity, and that yields per hectare decline significantly with land size. This is consistent with poor quality land and a labour constraint in agricultural activity: there is in-sufficient return on labour to justify further investment of effort to cultivate the land, given the typically low-input, labour-intensive farming methods. Food shortage models also show that adult labour is relatively unproductive: having more adults in the household produces little reduction in the incidence of food shortages. Overall the results suggest some directions for development interventions, by highlighting that the critical issue for economic and social development among these communities is to improve the productivity of both land and labour.