Nice project man) I think you should read this article, cause it may help you to make tracking more accurate and without manual labling
https://ai.googleblog.com/2018/06/self-supervised-tracking-via-video.html