The most expensive lot of New York's November auction season may not reach the rostrum. Days before Sotheby's is due to offer Vincent van Gogh's Chataigniers en fleurs, an 1890 canvas carrying a $180 million estimate, heirs of its former owner have demanded that the painting be withdrawn, arguing that it changed hands in 1936 only because of Nazi persecution.

The heirs are descendants of Samuel Fischer, the German Jewish publisher who bought the work, known in English as Blossoming Chestnut Trees, from a Paris gallery in 1909. Their Berlin lawyer, Gunnar Schnabel, wrote to Sotheby's restitution chief Marco Toracca on 12 November, asking the house to show that the 1936 sale took place at market value and would have happened even without Nazi rule. The letter, seen by the New York Times, warned that otherwise no auction should take place, while adding that the family was ready to seek a fair settlement under the 1998 Washington Principles.

According to the heirs, the Fischer publishing house suffered heavy losses after the Nazis came to power, and the painting was sold in Amsterdam in 1936, three months after the family had left Berlin for Vienna. Sotheby's own published history traces a different emphasis: the canvas was left with the dealer Paul Cassirer in Amsterdam in 1932 or 1933, passed to Brigitte and Gottfried Bermann-Fischer after Samuel Fischer's death in 1934, and moved to Henriette and Marcel Kapferer in Paris in 1936. The present Argentine owners acquired it in 1975.

Sotheby's said it had carried out extensive research into the painting's ownership and was in dialogue with the family's representatives, adding that it was committed to working toward a mutually satisfactory resolution. The house has not said whether the lot will still be offered on Wednesday 18 November as planned. The painting is the centrepiece of the Blaquier-Arrieta collection, twelve works estimated together at about $450 million, and it is backed by a Sotheby's guarantee. At $180 million it would be the highest estimate of any lot this season and, if sold, would set a new auction record for Van Gogh.