When we use 'a large number of' we use a plural verb after it.
A large number of people do this.
When we use 'the large number of' we use a singular verb after it.
The large number of people does this.
What about the case when we use 'a large number' without 'of'?
A large number do it.
OR
A large number does it.
It depends on the context.
If we say
A large number do it
we are implying, from context, an "of X" where X is plural. For example
Many people eat peanuts. What's more, a large number [of people] do it.
Or using the question/answer approach from lmus which reads much better:
How many people eat peanuts? A large number do it.
if we say
A large number does it
we are talking about a singular "large number". For example
One thousand million dollars is the target for the telethon. A large number does it! [makes the telethon reach its target]
Or (using Q&A)
What number makes the telethon reach its target? A large number does it!
I hope the examples make sense
What would convince the owners to keep the playground open?
A large number of visitors does this.
Who highfives the janitory at the playground?
A large number of visitors do this.
In the first example "A large number of visitors" is taken as one whole thing and used as the singular thing that convinces the owners to keep the playground open.
The second example talks about the individual visitors that highfive the janitor, and there are a lot of them.
To fully answer your question: it's not the "of visitors" that decides if it's singular or plural. In fact the "of X" part can be dropped if it is clear from context what the large number is "of" without changing anything else about the sentence.
What would convince the owners to keep the playground open?
Kids! A large number does this.
Who highfives the janitory at the playground?
Kids! A large number do this.
(thanks to @ParibusCeteris for providing these examples)
