Checks if a subset is a subgroup of a group.