Soft materials - foams, emulsions, pastes and granular media - can act as unusual solids, peculiar liquids and much more in between. A tremendous recent effort has uncovered how changes in the constitution (e.g., packing density) lead to a rigidity loss, or jamming, transition. A marginal point separates the rigid, or jammed, state from the freely flowing/vacuum/unjammed state, and rigidity is defined as resistance to deformation in linear response in this picture. Here I propose a radically different perspective: Because marginal materials are on the verge of 'falling apart', linear response becomes irrelevant at the marginal point. I claim that the surprising consequence of the marginal point is that it spawns a whole range of intrinsically nonlinear extreme phenomena, such as shock waves, yielding, flow, plasticity, rearrangements, strain softening and strain stiffening. Normally, one has to drive hard to reach extreme physics, but even a gentle touch is sufficient to drive marginal matter outside their vanishingly small linear response regime. Combining experiments and simulations I will uncover the extreme physics of marginal matter. I have identified weakly compressed grains, wet foams, and weakly connected elastic networks as prime examples of experimentally relevant and realizable marginal materials. Highly organized rearrangements, nonlinear elasticity and flow, and shock waves are their dominant extreme excitations. These intrinsically nonlinear phenomena are poorly understood and are not captured by the linear response/jamming paradigm. I believe they lie at the heart of much of the physics of driven soft matter, even away from the marginal point - and the marginal limit brings out the essential physics and provides a much needed organizing principle for these extreme excitations. By squeezing, shearing and kicking the frailest granular packings, foams and elastic networks I can make, I ultimately aim at providing a new understanding of the physics of real soft materials.