libvpx

Change tx_size_search_method setting for non-rd speed 5.

Use the same settting as in speed >=6.
This will use same logic for tx_size selecton as in speed >=6,
which limits the transform size and reduces ringing artifact.
Also metrics go up on average with this change: ~2% for PSNR, ~10% for SSIM.

Change-Id: Ia2d50db236ae1cc72f742bfa6c9ec5ea50ff0e0a