For the fruit topping:
3 tablespoons (1 1/2 ounce) unsalted butter
1/2 cup (3 ounces) packed light brown sugar
1 1/2 pounds ripe but firm nectarines (about 3-4 large, or 6 medium)

For the cake:
8 tablespoons (4 ounces/1stick) unsalted butter, softened to room temperature
1/2 cup (4 ounces) granulated sugar
seeds from 1 vanilla bean (bean split lengthwise, seeds scraped out, pod saved for another use)
2 tablespoons finely chopped fresh rosemary
1 teaspoon vanilla extract
2 large eggs, at room temperature
3/4 cup (4 ounces) sweet white rice flour
1/4 cup (1 1/4 ounces) millet flour
1/4 cup (1 ounce) oat flour
1/4 cup (1 1/4 ounces) stone-ground yellow cornmeal
2 tablespoons (1/4 ounce) tapioca starch
1/2 teaspoon xanthan gum
2 teaspoons baking powder
1/4 teaspoon salt
1/3 cup (2 1/2 ounces) crème fraîche (or sour cream), plus extra for serving
1 cup chopped nectarines, from above