Many groups that arise naturally are groups of symmetries: for example, the Orthogonal group is the group of linear transformations of Euclidean space which preserve distance. Many interesting groups are groups of matrices. These often turn out to be Lie groups: that is, they have an interesting and rich geometric structure.
Finite groups can be well understood by looking at representations. Much of this theory carries over to compact groups, although a bit of topology and analysis is needed. If you have compact Lie group, you can say even more, as the geometric structure allows you to parameterise the representations.
A 3rd year project would look at matrix groups, exploring carefully
some classical examples, and might look at some representation theory,
or some basic Lie theory. You should have good marks in, and have
enjoyed, the courses "Group Theory", "Linear Algebra 2" and "Rings,
Polynomials and Fields". Some interest in geometry and basic analysis
would help. Recommended books are:
"Matrix groups for undergraduates" by K. Tapp
"Naive Lie Theory" by J. Stillwell
A 4th year project would look (much) more at the representation
theory; my personal taste is towards the theory of general compact
groups, but the project could concentrate upon Lie groups. You should
have a good background in Analysis, and have enjoyed the course
"Topology", and perhaps also "Algebras and Representations" and/or
"Hilbert Spaces and Fourier Analysis". Recommended books are:
"Representations of compact lie groups" by T. Brocker and T. tom Dieck
"Compact Lie groups" by M. Sepanski.