Yes, you can use your HD camera with ANY-maze, although you may wish to adjust the image to a lower resolution, which you can easily do from within ANY-maze.
If your camera shows just a single piece of apparatus and the animal is not very small in comparison to the apparatus (for example, a mouse on a typical plus-maze is not very small, but a mouse in a 4mx4m open field would be) then there’s really no benefit to using high definition images and all it does is require the computer to do much more processing (for example a Full HD image of 1920 x 1080 requires nearly 7x as much processing as a 640 x 480 image).
However, high definition images can be very helpful if you intend to use a single camera to view multiple apparatus. For example, if you laid out four mouse open fields in a square (i.e. with one open field in each quadrant) then you could set up a single Full HD camera above them and ANY-maze would see an image of 960 x 540 of each open field, which would be ideal for tracking.