I think what you might want to try is content aware fill where your filling the newly added transparent border. If that is possible. Don't know how well it would work. For it sound like you don't want to scale the frame. That your actually trying to protect them with an alpha selection. I also think that content aware scaling each frame would produce a very jittery video.
However I think the best way would be to just resize video the fill the aspect ratio area size you need, Crop to the required aspect ratio then resize. You will loose a borders on the top and bottom or the sides when you change aspect ratios. Something like you see on TV this film has been modified to fit your screen. Or you could distort the video resize the video to the new aspect ratio size without constrain.