The music box is one of many pieces of art that Neal Caffrey has been associated with during his criminal career. It belongs to the Russians, but was stolen by the Nazis during World War II. Years before Neal was imprisoned, he and Alex Hunter went looking for it at the Amalienborg Palace in Copenhagen. Neal and Alex snuck into the palace and hung out with the Royal family while they were scouting for the box's location. They got close and almost had it, but they were forced to abandon the heist, presumably in order to escape the palace guards.
The box is revealed to carry the equation to a fractal antenna, embedded into the music it plays. Alex tells Neal that her grandfather, Gerhard Wagner, encoded a sunken U-boat's final SOS antenna design into the music box. His plan was to come to America, build a receiver, and find the boat. However, he fled Germany after the war, losing the box in the process, though managing to keep one piece of the music box. On his deathbed, he passed the search for the music box to Alex and gave her the piece he had.
According to Mozzie, 'the last credible report has the box disappearing from Königsberg Castle in Germany in 1945.' He also insinuates that the box had been whole until its disappearance, matching Alex's story about her grandfather's history with the music box.