Gather all your ingredients. Preheat your oven to 350°F (175°C). Grease and flour a 9×5-inch loaf pan or line it with parchment paper for easy removal.

In a medium bowl, whisk together the flour, baking soda, baking powder, cinnamon, nutmeg, ginger, and salt. Set aside.

In a large mixing bowl, beat the softened butter, brown sugar, and granulated sugar together until light and fluffy, which should take about 3-4 minutes.

Add the eggs one at a time, ensuring to mix well after each addition. Stir in the vanilla extract until fully combined.

Gradually add the dry ingredient mixture to the butter mixture, alternating with the buttermilk. Mix until just combined.

Gently fold in the diced apples and chopped dates until they are evenly distributed throughout the batter.

Pour the batter into your prepared loaf pan, ensuring to smooth the top with a spatula for an even bake.

Bake for 50-60 minutes, or until a toothpick inserted into the center comes out clean.

Allow the cake to cool in the pan for about 10 minutes. Carefully turn the cake out onto a wire rack to cool completely.

In a saucepan over medium heat, combine the granulated sugar and butter. Stir until melted and at a rolling boil.

Allow to boil without stirring for about 3-4 minutes, or until the mixture turns a light amber color.

Remove the saucepan from the heat and stir in the heavy cream, vanilla extract, and a pinch of salt.

Let the toffee cool slightly before drizzling it generously over the cooled cake. Allow the toffee to set before slicing and serving.