If your subgroup H is maximal in the group G then you can click here in order to find the possible indexes and transformation matrices.
If your subgroup H is not maximal in G and you know
the index G:H = n then one have to construct the
lattice of maximal subgroups from the group to the desired subgroup. Then
using the transformation matrices for each (already) maximal subgroup
you can find the transformation matrix. Of course there
are a lot of different chains of max subgroups. All this work can
be done clicking
here.
The knowledge of index and/or transformation matrix is mandatory.
Finally if you don't know the index you can draw the group/subgroup
leattice which have to help you to figure out the index of subgroup. In order to
construct and draw the lattice of maximal subgroups click
here.