The Federal Reserve is quietly preparing a fundamental rethink of how it provides core banking services, from the price of processing checks to the way it supervises the smallest community lenders.