If there is one job we carry out as gardeners that creates as much controversy as any, its clearing autumn