Peel fresh peaches or nectarines, if desired. Pit and slice peaches or nectarines.
For sauce, in an 8-inch skillet stir together the brown sugar, water, brandy or rum, margarine, lemon juice, and mace or nutmeg. Bring to boiling, stirring occasionally.
Add fresh peaches or nectarines or frozen peaches to the sauce; stir gently. Reduce heat. Simmer, covered, about 5 minutes for fresh peaches or about 10 minutes for frozen peaches or till tender.
Place a 1/4-cup scoop of ice milk into each of 4 dessert dishes. Arrange peaches around ice milk; spoon sauce over top.