How do you call the daughter of your brother-in-law?

We don't have a specific Word for this kin relationship in English. If you think of your daughter-in-law as being like your own daughter, then her sister would be like your daughter too, and in that case the sister's son would be like your grandson. I have never heard anyone use the term "grandson-in-law," but it's possible to use such a Word. I would use it if I had a relationship with this boy (in some families the mother-in-law might not ever even meet this relative). Otherwise, "daughter-in-law's sister's son" is the only way to precisely describe it.