@eviraThe audacity of how the data exposes a real gap in reasoning claims when you actually look at the numbers. Astra hit 88% on the INDUCTION benchmark and nearly saturated the task while Fable 5.1 sat at just 33%. This performance data comes from a single batch run at xhigh thinking effort though a residual batch is still running for non-evaluable items which may increase final numbers. The cost side gets uglier fast. Fable 5.1 consumed 32 million output tokens across four runs to generate 66 successful API responses while Astra cost approximately one-quarter of the total price incurred by Fable 5.1. It really highlights a gap in reasoning claims between models since Astra's lower cost comes with higher efficiency relative to success rate.

@eviraThe audacity of how the data exposes a real gap in reasoning claims when you actually look at the numbers. Astra hit 88% on the INDUCTION benchmark and nearly saturated the task while Fable 5.1 sat at just 33%. This performance data comes from a single batch run at xhigh thinking effort though a residual batch is still running for non-evaluable items which may increase final numbers. The cost side gets uglier fast. Fable 5.1 consumed 32 million output tokens across four runs to generate 66 successful API responses while Astra cost approximately one-quarter of the total price incurred by Fable 5.1. It really highlights a gap in reasoning claims between models since Astra's lower cost comes with higher efficiency relative to success rate.