I would think 1 to 4 percent Neanderthal in most Modern Human DNA would be enough to make some differences since we don't know what the vast majority of the human genome does specifically.
Since the overall difference between the chimpanzee and Human genome is only about 3 percent in total and seeing as how just minute differences of a percent in the human genome give us the visible differences in eye, hair and skin color, height, skeletal formation and also, ability to taste different substances and food and chemical intolerances among people of different regions I would imagine that some of the things we consider human already are actually Neanderthal in origin in some of the population.
One interesting thing is that people have the most variation in visible traits alone, amongst themselves and Sub-Saharan Africans, in areas where we interacted with neanderthals the longest i.e.- Southern Europe and the Middle East. I've heard some suggest that red hair and green eyes come from Neanderthals but then I also heard it comes from evil space lizards doing nasty chemistry with the Sumerians.
I suspect the question of What made us human as a whole will never be fully answered without the time machines of evil space lizards and a full blooded Sumerian that can use the technology.
