If 2 identical twins male each have offspring with other 2 identical twins? Will the baby of both look alike?

Not necessarily. Since each sperm and egg contain different combinations of genetic material, the kids would vary in looks. An event called chromosomal crossover is a big factor too. The kids could look related but will most likely all look different, similar to the way siblings from the same parents look different.

Well even though people look a like they may have certain small differences. and who knows if the baby of each couple wil take on more of the mothers or fathers looks. Maybe the baby of one couple could have brown hair like the mum and blue eyes like the dad but the other baby could be completely oposite. You can't really tell how the baby will look

Each organism that goes through sexual reproduction inherit both recessive and dominant alleles. Who knows what genotype an organism inherit. Some may have the phenotype of the recessive alleles which the twins have. It all depends on probability and it is highly unlikely that they will look the same.

The genetic constituents of twin are different and as such even if a twin marries another twin, their kids might just appear differently.

Possibly but one baby set could look more like their mom, the other like their dad.

They won't look any more alike than you and your siblings.