Collision deformer

I was looking into it at one point and these are some of the resources I remember off the top of my head.

https://www.highend3d.com/maya/plugin/jlcollisiondeformer-for-maya - You can look at the source code http://oaktrust.library.tamu.edu/bitstream/handle/1969.1/155037/WANG-THESIS-2015.pdf?sequence=1

What it basically boils down to is finding intersections between meshes and then calculating how to respond to those intersections. A good thing is that a lot of game developers have struggled with this problem, so there are a lot of topics on forums around the web.

The way I've done it in the past was, finding intersections by casting a ray from the vertex along it's normal and checking how many intersections with the collider mesh there are. It works only for closed geometries. If there is an odd number of intersections that means that the current vert is inside of the collider geometry.

Then for responding, to it, I used three methods for different cases

1. casting a ray from the vert in the negative normal direction and finding the first intersection with the collider. That becomes my deformed vert position.
2. finding the closest point to the vert on the collider mesh.
3. using the first method I find the intersected face of the collider and then I project the current vert along that face's normal to get an intersection with the collider.

The 3rd method had best results most of the time, but it was computationally more expensive.

Hope that helps!

Oh, and recently a colleague showed me the membrane node in Maya, which acts as a collision deformer, if you want to just have a play with something and maybe compare your results, though, im not sure how much you should trust it to be a good one.