rotX and rotY are relative to the vertices' positions, to rotate all points about. It doesn't matter what the x and y are, translation is applied after. So...
vertices = [{x:2, y:0}, {x:4, y:2}, {x:2, y:4}, {x:0, y:2}]
rotX = 0, rotY = 0, angle = 0, x = 0, y = 0
=======
R . 0 . .
. . . . .
3 . . . 1
. . . . .
. . 2 . .
rotX = 2, rotY = 2, angle = Math.PI, x = 0, y = 0
=======
. . 2 . .
. . . . .
0 . R . 3
. . . . .
. . 1 . .
rotX = 2, rotY = 2, angle = Math.PI, x = 2, y = 2
=======
. . . . . . .
. . . . . . .
. . . . 2 . .
. . . . . . .
. . 0 . R . 3
. . . . . . .
. . . . 1 . .
If you are using ancient GL-style translatef- and rotatef-style calls, you would need to do something like this:
glTranslatef(group->x, group->y);
glRotatef(group->angle, group->x + group->rotX, group->y + group->rotX, -1.0f);