When two hypotheses are equally adequate to the data, and equal in predictive power, extra-theoretical criteria for choosing between them might come into play. They include not just questions about best fit with other hypotheses or theories already predicated to enquiry, but the aesthetic qualities of the competing hypotheses themselves—which is more pleasing, more elegant, more beautiful?—and of course the question of which of them is simpler.
Simplicity is a desideratum in science, and the quest for it is a driver in the task of effecting reductions of complex phenomena to their components. It lies behind the assumption that there must be a single force in nature, of which the gravitational, electroweak and strong nuclear forces are merely manifestations; and this assumption in turn is an instance of the general view that there might ultimately be a single kind of thing (or stuff or field or as-yet-undreamt phenomenon) out of which variety springs by means of principles themselves fundamental and simple.
Compelling as the idea of simplicity is, there is no guarantee that nature itself has as much interest in simplicity as those attempting to describe it. If the idea of emergent properties still has purchase, biological entities cannot be fully explained except in terms of them, which means in their full complexity, even though considerations of structure and composition are indispensable.
Two measures of complexity are: the length of the message required to describe a given phenomenon, and the length of the evolutionary history of that phenomenon. On a certain view, that makes a Jackson Pollock painting complex by the first measure, simple by the second; while a smooth pebble on a beach is simple by the first and complex by the second. The simplicity sought in science might be thought to be what is achieved by reducing the length of the descriptive message: encapsulation in an equation, for example. But: could there be an inverse relationship between the degree of simplicity achieved and the degree of approximation that results?
Of course it would be nice if everything in the end turned out to be simple, or could be made amenable to simple description. But some things might be better or more adequately explained in their complexity—biological systems again come to mind. Resisting too dissipative a form of reductionism there might ward off those silly kinds of criticism claiming that science aims to see nothing in the pearl but the disease of the oyster.