Converting 12 bit color values to 8 bit color values C++

I'm attempting to convert 12-bit RGGB color values into 8-bit RGGB color values, but with my current method it gives strange results.

Logically, I thought that simply dividing the 12-bit RGGB into 8-bit RGGB would work and be pretty simple:

// raw_color_array contains R,G1,G2,B in a bayer pattern with each element
// ranging from 0 to 4096
for(int i = 0; i < array_size; i++)
{
    raw_color_array[i] /= 16; // 4096 becomes 256 and so on
}

However, in practice this actually does not work. Given, for example, a small image with water and a piece of ice in it you can see what actually happens in the conversion (right most image).

Why does this happen? and how can I get the same (or close to) image on the left, but as 8-bit values instead? Thanks!

EDIT: going off of @MSalters answer, I get a better quality image but the colors are still drasticaly skewed. What resources can I look into for converting 12-bit data to 8-bit data without a steep loss in quality?