Problems with Damson and Plum Trees

I have 2 damson and 2 plum trees. They are 6 years old. They have grown fairly profusely but little or no fruit . Only a single damson and a single plum last year. I have planted a new damson and plum tree last year of different variety in case pollination problems. There is an old damson tree close by and it bears fruit each year . What do you think is wrong?