Computing

You could just put each MP4 into a file and go to right click - > Properties - > Customize - > Change Icon.

Or, you can change the MP4 icon by going to:
Start -> regedit.exe -> HKEY_CURRENT_USER -> Software -> Microsoft -> Windows -> CurrentVersion -> Explorer -> FileExts -> .mp4 -> UserChoice

Then look at the value next to "Progid" and underneath "Data". For instance, mine would say gomplayer.mp4.

Then go to HKEY_CLASSES_ROOT and search for your specific file name that we just found. Once you find it, open up DefaultIcon. If there is no folder called DefaultIcon, you can right click on the folder and go to New -> Key. Name it DefaultIcon.

Once inside the DefaultIcon folder, you can then edit the value next to (Default) and underneath Data. Change the directory to your image file. For instance, if I had a folder on my desktop named MyFolder, with an image file such as MyIco.ico inside, the directory would be:

C:\Users\Taylerzy\Desktop\MyFolder\MyIco.ico, 1

The one at the end tells which icon number we are referring to (alphabetically).
http://www.tomshardware.co.uk/answers/id-1724369/change-icon-mp4-file.html