Only the fuel pressure regulator can cause that. Liquid doesn't compress. So when the injectors open, pressure falls. When they close, pressure rises. Only the pressure regulator can react to the pressure change and on most engines the pressure is pretty smooth, but on your engine it isn't. Only one component can influence pressure change and react to it. The fuel pressure regulator.
Way back, early EFI systems like Bosch had accumulators in them to keep system pressure steady, and hold a reserve of pressure to assist with hot restart. Modern systems rely on the limited volume within the pressure regulator to act as the accumulator. The diaphragm depresses the spring for a bit of volume before it opens at the relief pressure. That volume is the accumulator portion. That's why pressure holds after turning off the pump. Once the regulator spring has completely extended, there is no more pressure.
When evaluating pressure in liquids, it is important to keep in mind that liquids don't compress. So any residual pressure is being applied by another source. Like the spring in the pressure regulator. It's not like air in a tire. Air compresses.