Measurements of branching fractions, polarizations, and direct CP-violation asymmetries in B ---> rho K* and B ---> f0(980) K* decays