"Add color to palette" setting the default palette

I usually have 3-5 palettes open for any given project, and the Fill Color Dialog and Outline Color Dialog have "add color to palette" drop downs. So far so good. The frustrating part is that there's a default selected for which palette gets the color if I just click on the button instead of using the drop down. It's (usually) not the palette I've set as the default palette more generally, and it seems to be picked at random - and I can't find any way to select which palette gets the "add color to palette" default.

So my question is, how do you select which palette is set as the default in the "add color to palettes" drop downs?