n.— «Among country and bluegrass musicians, Mr. Martin is regarded as a master of a difficult five-fingered playing style known as clawhammer or frailing, in which the instrument’s strings are pushed down by fingernails, rather than pulled up with picks.» —“Jokes and Films Are Fun, but He Loves His Banjo” by Dave Itzkoff New York Times Feb. 2, 2009. (source: Double-Tongued Dictionary)
The term is used in Pete Seeger’s instruction manual for banjo, first published in 1948. It may be a good bit older than that; you can hear the style it refers to on records from the 1920s and ’30s, but I can’t document that it was called that at the time.
Also, FWIW, purists differentiate between the frailing and clawhammer styles, but that’s probably beyond the scope of this comment.