Butter and flour a 9-inch spring form pan. In a small bowl, mix the cake flour with the baking powder and salt.
In a bowl of a standing mixer fitted with a paddle, beat the sugar with the almond paste until crumbly. Add the butter and beat t high speed until light in color and fluffy, about 2 minutes. Add the eggs, one at the a time, beating until fully incorporated between additions. Beat the vanilla extract and vanilla seeds. Gently fold in the flour mixture until fully incorporated.
Scrape the batter into the prepared pan. Arrange the plums over the top of the batter. Bake for 1 hour and 5 minutes, or until the cake is deeply golden and a toothpick inserted in the center comes out clean.
Transfer the cake to a rack and let cool for 15 minutes. Run a knife around the edge of the cake and remove the outside ring of the pan. Let the cake cool for at least 30 minutes longer. Serve warm or at room temperature topped with crème fraiche and slivered almonds.
Make ahead: The cake can be made one day ahead and stored in an airtight container at room temperature.