You are absolutely correct, technically any of the minor modes would work (dorian, phrygian or aolian). It's just that a minor seventh chord by itself doesn't really sound phrygian or aolian by nature.
Phrygian: A more phrygian sounding chord would be a susb9 more so than a plain old min7. Dorian also leaves the chord player more choices, with the phrygian mode a min6, min9, min13th are out of the question but the dorian mode gives you all those chords.
Aolian: Seems to work better in progression than static vamps. In the case of Dmin, a Dmin7 - Bb - C type chord progression may work better.
Some fun improv ideas for dorian, as an example, D dorian. Try these scales and arpeggios (extentions in parenthesis):
D dorian (Cmajor) (1 2 b3 4 5 6 b7)
D minor pentatonic (1 b3 4 5 b7)
E minor pentatonic (2 4 5 6 1)
A minor pentatonic (5 b7 1 2 4)
F triad (arpeggio) (b3 b7 9)
G triad (arpeggio) (4 6 1)
C triad (arpeggio) (b7 9 4)