Free access
Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms

The simplex method is strongly polynomial for deterministic Markov decision processes


We prove that the simplex method with the highest gain/most-negative-reduced cost pivoting rule converges in strongly polynomial time for deterministic Markov decision processes (MDPs) regardless of the discount factor. For a deterministic MDP with n states and m actions, we prove the simplex method runs in O(n3m2 log2 n) iterations if the discount factor is uniform and O(n5m3 log2 n) iterations if each action has a distinct discount factor. Previously the simplex method was known to run in polynomial time only for discounted MDPs where the discount was bounded away from 1 [Ye11].
Unlike in the discounted case, the algorithm does not greedily converge to the optimum, and we require a more complex measure of progress. We identify a set of layers in which the values of primal variables must lie and show that the simplex method always makes progress optimizing one layer, and when the upper layer is updated the algorithm makes a substantial amount of progress. In the case of nonuniform discounts, we define a polynomial number of “milestone” policies and we prove that, while the objective function may not improve substantially overall, the value of at least one dual variable is always making progress towards some milestone, and the algorithm will reach the next milestone in a polynomial number of steps.

Formats available

You can view the full content in the following formats:

Information & Authors


Published In

cover image Proceedings
Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms
Pages: 1465 - 1473
Editor: Sanjeev Khanna, University of Pennsylvania, Philadelphia, Pennsylvania
ISBN (Print): 978-1-611972-51-1
ISBN (Online): 978-1-61197-310-5


Published online: 18 December 2013




Research supported by NSF grant 0904325. We also acknowledge financial support from grant #FA9550-12-1-0411 from the U.S. Air Force Office of Scientific Research (AFOSR) and the Defense Advanced Research Projects Agency (DARPA).

Metrics & Citations



If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

There are no citations for this item

View Options

View options


View PDF

Get Access







Copy the content Link

Share with email

Email a colleague

Share on social media