The unnatural glare of the fire across the dark water was seen from the Long Island and Connecticut shores. The flames shot up in huge columns, lighting the water for miles around.

William Sidney Mount, an artist of some renown for his paintings of Long Island country settings, witnessed the calamity and described how local mariners struggled to sail through pack ice clogging ports and inlets. Fishermen, thinking they might rescue victims while the steamboat was only two miles away, set out from their harbor in the bitter cold. But just when they thought they were within hailing distance of the flaming wreck, the winds and the tides shifted, sweeping the Lexington back into the middle of the Sound. Defeated by the whims of nature, the intrepid fishermen had no choice but to return home, the water being too rough for them to venture into the Sound.

Captain William Tiffell, of the sloop Improvement, sighted the burning pyre, but failed to offer assistance, claiming that he thought the steamer had her boats, and he was afraid that if he stopped he would lose the tide coming into the harbor. Like Captain Stanley Lord of the California seventy-two years later, who was accused of standing by while the Titanic sank, Terrell was denounced as a cruel and heartless man. Because of his alleged indifference to the suffering of the passengers on the Lexington, he came within an inch of having his master's papers revoked. But later studies showed that he was a good twelve miles distant and facing a contrary wind. Investigators considered it doubtful that he could have reached the stricken vessel in time to save its ill-fated passengers had he tried.

Discovering the steamer on fire, Captain Oliver Meeker of the sloop Merchant tried to sail his vessel from the pier at Southport.

But the combination of a shallow harbor and a falling tide caused the Merchant to run aground.

Captain Hillard and Benjamin Cox had drifted about a mile from the Lexington when she went down. A scattering of clouds strayed over the mainland and a bright moon illuminated the Sound. The night air was incredibly cold, and the men tried to keep warm by whipping their hands and arms around their bodies. They were as miserable as two humans could get. Then, as if ordained to multiply their agony, a large swell overturned their cotton bale. Plunged into the frigid water, Millard and Cox struggled to climb on the opposite side. Losing the paddle was a double blow. Besides employing it as a means to keep warm through exercise, they had found it useful for steering against the tide. Now the bale became uncontrollable and rolled heavily under the onslaught of the waves.

Cox had abandoned the boat wearing only a flannel shirt, loosefitting pants, boots, and a cap. An old mariner, Hillard had wisely worn his heavy woolen pea jacket. He gave Cox his vest, and then rubbed the passenger's arms and legs, beat him on the body, and made every attempt to keep his blood circulating.