Under an experiment in the nation's courts, states are using secretive, psychological tools to predict which inmates will reoffend.