In a large bowl, whisk together the pudding mix with the milk for 2 minutes. Set aside for 5 minutes to thicken.
Line the bottom of a 9x13-inch pan with graham crackers (you may need to use halves or quarters to fill in spots).
Spread half of the pudding over the layer of graham crackers.
Add another layer of graham crackers on top of the pudding, then spread with the remaining pudding.
Top with a final layer of graham crackers and spread the Cool Whip evenly over top.
Crush 1 graham cracker into crumbs and sprinkle over the Cool Whip. You can also top with fruit at this time instead of graham cracker crumbs.
Cover and refrigerate until cold and set, about 3 hours, or overnight. Leftovers should be stored, covered, in the refrigerator for up to 3 days.