In the bowl of a stand mixer fitted with the paddle attachment, beat together the butter and sugar at medium speed until combined, light, and fluffy, about 1 minute. Add the eggs, one at a time, mixing well after each addition. Reduce the mixing speed to low, and alternate between adding the flour mixture and the milk in two increments. Add the plum juice, bourbon, lemon zest, and vanilla extract and mix until incorporated, taking care not to over mix. Fold in the finely chopped fresh plums by hand.