Trading firms, brokers and some academics have doubts about a central pillar of the SEC’s plan to overhaul stock-market structure: auctions for small investors’ orders to buy and sell stocks.