Preheat your oven to 350°F (175°C). Place a 10-inch cast iron skillet in the oven to heat up while you prepare the batter.
In a stand mixer bowl, combine the granulated sugar with the finely grated zest of one orange. Set this mixture aside.
Trim the ends off both oranges, then cut them into 8 pieces each. Puree the oranges in a blender or food processor until smooth. You'll need about 1 1/2 cups of puree.
To the bowl of sugar and zest, add the vanilla extract, fennel seeds (if using), and a pinch of salt. Mix these together vigorously by hand.
Once the sugar mixture is fragrant, add the room temperature butter to it. Cream together the butter and sugar mixture on medium-high speed until fluffy, which should take about 5 minutes.
Next, add the room temperature egg yolks to the butter mixture and beat until well incorporated, which should take around 1-2 minutes.
Whisk together the all-purpose flour, and, if using, semolina flour, baking powder, and salt in a separate bowl.
Scrape down the sides of the mixing bowl and paddle, then stir the batter to ensure it's well mixed.
Slowly incorporate the reserved orange puree into the batter on medium-low speed, then increase to medium-high to blend well.
Gradually add the dry ingredients to the mixer while it's on low speed. Increase the speed to medium-high, scraping down the bowl and paddle until the batter is thoroughly mixed.
Stir in the chopped toasted walnuts until they are evenly distributed throughout the batter.
Carefully remove the hot cast iron skillet from the oven and brush it generously with olive oil. Pour the batter into the skillet, ensuring it is spread evenly across the surface.
Bake the cake for 35 to 45 minutes, or until the cake is set in the middle and golden brown on top. You can check for doneness by inserting a cake tester into the center. It should come out clean when the cake is done.
Allow the cake to cool in the skillet for about 30 minutes before serving. Enjoy warm or at room temperature.