1 - many people do prefer to use a 4S as it provides a bit of extra power to make it more responsive, but it does reduce flight times slightly since the 4s is heaver. The 5100 3s that we sell is not suitable for the Y6 since its discharge rate is too low. You will need something with an output of at least 60A. You can work out the current output by multiplying the capacity by the battery C rating. So for the 5100Mah 10C 3S Tattu battery, the current output is (5.1*10 = 51A) which is too low for a hexacopter, it is good enough for a quadcopter setup though. For more details check out our guide here http://www.dronetrest.com/t/lipo-batteries-how-to-choose-the-best-battery-for-your-drone/1277
2 - Yes if you want to use other motors/ESC just buy the frame on its own and choose whatever ESC/motors you want. - http://www.unmannedtechshop.co.uk/arducopter-y6-frame/. However the Y6 kit already comes with the 850Kv motors.
If you have any other questions about the 3DR Y6 please let me know and will be happy to help.