There is demand for an assessment tool that can assist a community to prepare, respond and recover from emergencies and disasters. A participatory approach, in which communities undertake their own assessment, has been promoted as an ideal method of identifying strengths and vulnerabilities within the local community and has been associated with substantial co-benefits that directly enhance resilience. In this study, five small rural communities exposed to frequent flooding events evaluate the usefulness and efficacy of the Community Disaster Resilience Scorecard Toolkit Scorecard. Our participants reveal that the Scorecard covers relevant topics and has directly supported disaster resilience planning. Despite significant difficulties in attracting people to participate in the assessment, all of the participants considered that the process of assessment was immensely valuable. Overall the communities identified that the Scorecard does require adaption to provide a more accurate depiction of their community’s resilience and have suggested ways to achieve this. However underpinning the assessment and the evaluation tasks were deep concerns about the ‘shared responsibility’ position of government and the reliance upon fatigued volunteers to provide local knowledge and skills.