An insulating layer of a polyester microfleece or wool vest or shirt will help keep out the chill on walks below 40 F. I prefer microfleece as the insulating layer as it is thin enough to fit under my regular waterproof jacket without bulk, and it dries fast if I get sweaty.
Why wear an extra layer rather than just wearing a heavier jacket? You are likely to warm up as you wall. And if you are dressed in layers you can take off the insulating layer and stow it, while still wearing your jacket to keep out the wind and rain.