Brady had a penchant for jewels, especially diamonds and had a collection worth over $2million dollars. Brady was a gregarious man and enjoyed Broadway. He was also known for his excessive eating habits, though not a large man. After his death, an examination showed that Brady's stomach was six times the size of an average person.
He was also the first person to own a car in New York City in 1895.
Brady was the inspiration for a 1935 film written by Preston Sturges entitled Diamond Jim and might have inspired a character called "Big Jim" in the Bob Dylan song, "Lily, Rosemary and the Jack of Hearts."