Well, first of all, you're assuming that Quirrell is an accomplished legilimens but since he was carrying someone around who is known to be one that is a fair assumption.

However, there are several indications in the books that giants were resistant to magic. As explained Harry Potter fandom page about giants:

Gifted with overwhelming raw strength proportionate to their prodigious size, they are difficult to detain by wizards since most spells tend to have little to no effect upon them [...]

The same site states that half-giants share this resistance (and we know that Hagrid does):

They also retained the natural resistance to most types of magical attacks from their giant blood.

So, since Legilimency is magic, I see no reason why Hagrid would not be as resistant to it as he is to other spells, like stunning, a spell we have seen him ignore when the Aurors were attempting to arrest him in The Order of the Phoenix. Therefore, his resistance would have stopped Quirrell from using Legilimency on him.