Why do you not assume you are looking at two different hot air balloons very close to one another? Weirdly, you know they're the same thing precisely because you know they're the same colours (despite the fact that they don't actually appear to be the same colours at all).
This is because of an inbuilt process called 'colour constancy' that allows us to identify something as the same colour in a variety of different lighting conditions.
But even though it comes from certain things that are already inside our brain, it is still just one theory that guides our perception. What is most fascinating is that we need not perceive colour in this way! Say if you are a painter and you need to perceive colours as they actually do appear to be, so that you can reproduce them, one does this by changing the theory used to perceive colours. Allow me to give an example.
Below is an optical illusion that works by exploiting colour constancy.
I'll show you what your eyes actually look at to work out the colour, and then I'll show you how to ignore what you normally see, by using the theory of colour perception painters use to paint.
On the lower square, our eyes see the light grey shading on the top and decide that the lower square must be mostly in shadow. Your brain therefore tells you that it's really a lighter grey but looks dark because the light isn't hitting it. So you actually see it as lighter.
On the other hand, the top square appears to be tilted upwards, and so your brain assumes its in full light. Plus your eyes read the dark grey around the edges as shading, which is more verification that the top square is otherwise in full light. Your brain then tells you that the top square is actually darker than the other, but appears quite light because it has a lot of light hitting it.
Because you're brain sees them as being under different light conditions, it tells you to see them as two different shades of grey, a much darker and a much lighter.
The illusion is of course broken if you cover up the light grey shading in the middle that is the main reason your eyes think the square are under different lighting conditions.
But this isn't the only way one can change their perception of the squares. A painter will be used to understanding how to isolate colours ignoring the lighting conditions. You can try this, too. If you ignore the light and dark shading and just compare the main colour of both squares, after a moment you should be able to see they are the same colour without even having to cover the middle up.
Basically, depending on which theory you use to perceive the above, actually changes what you perceive.