AI is a hobby of mine, I love it!  I also hate it!  The secret to any predictive model is dealing with random walk.  Filter out the random events to uncover the patterns in the data.  It's been statistically proven that free and open markets are random, not just exhibiting randomness, but really random.  An RNN or any other kind of Neural Network shouldn't be able to predict it because what they do is predict the future based on patterns from the past and a random system doesn't have a pattern. Once you filter out the randomness there isn't anything left to latch onto.  

Anyone that cracks that particular nut has a Nobel prize waiting for them and probably a few billion dollars in licensing agreements coming.

Obviously you can.  This means that the crypto markets are not free and open, but have a driver of some kind.  I'd be interested in learning what the driver is or if you uncovered several drivers, or is it some pile on effect from algorithmic traders that might be negated if better tools come to the fore.  

For the record, I really hate dealing with TensorFlow, I'm impressed you could wrangle it.  You're getting an upvote from me just for that!