Around 1957, I was 12, , Marvin Rainwater came out with this song, and it is still one of my favorites.
I shall always find it fascinating how certain songs continue to evoke the feelings (or even the general state of being) I had at the point in my life that I first heard them. Music is extremely powerful stuff.
I had to go look, because I always thought it was a reference to the game of Jacks. You are correct in your RR explanation. I find reference to a 1913 song titled Ballin' the Jack, and then another reference to Danny Kaye and a song of that title, but I don't know if he sang that original song or if he wrote one of his own with that same title.