Preheat oven to 350. Grease a 9x13 pan.
In the bowl of a stand-mixer fitted with the paddle attachment, beat together the cream cheese with the orange zest, ginger, and powdered sugar until smooth, scraping sides of the bowl as needed. Set aside.
In a mixing bowl, whisk together the milk, pumpkin puree, eggs, butter, sugar, vanilla, and pumpkin pie spice until well combined.
Place half of the bread cubes in a layer in the greased 9x13 pan. Drizzle half of the pumpkin mixture over the bread. Using a rubber spatula, spread the cream cheese mixture over top of that (it doesn't need to be perfect). Then top with the remaining bread cubes and drizzle the remaining pumpkin mixture over top of everything. Press the bread down into the liquid to saturate. Sprinkle the top of the french toast with the pecans and finish with fine sea salt.
Bake 35-40 minutes, until cooked through in the center. Serve hot with pure maple syrup.