Preheat oven to 350 degrees (F).
Crush the graham crackers. (Easy tip: place the graham crackers in a ziplock bag and roll them to crumbs with a rolling pin.)
Mix the crumbs with the melted butter and brown sugar. Spread this on a baking sheet covered with foil or parchment paper and bake 10 minutes or until golden. Let cool, then break into crumbs again.
Meanwhile, whip the cream with the powdered sugar and vanilla. Set half of this aside for the topping.
Juice the limes and strain out any seeds.
Mix the condensed milk with the lime juice. The citrus will cause the milk to thicken. Gently stir in half the whipped cream mixture.
Layer some of the baked crumbs into eight parfait or martini glasses, then add some of the key lime mixture; repeat, setting aside a few crumbs for topping.
When all the ingredients are distributed, top with dollops of remaining whipped cream. Sprinkle remaining crumbs and zested lime of you want a stronger flavor.
Keep chilled until ready to serve.