Let the three terms of the GP be: ra,a,ar
Then,
ra+a+ar=26
a(r1+1+r)=26
Also, sum of squares:
r2a2+a2+a2r2=364
a2(r21+1+r2)=364
Now,
(r1+1+r)2=r21+1+r2+2(r1+r+1)
(a26)2=a2364+2(a26)
a2676=a2364+a52
676=364+52a
52a=312
a=6
Now, r1+1+r=626=313
3+3r+3r2=13r
3r2−10r+3=0
(3r−1)(r−3)=0
r=3 or r=31
Hence, the three terms are 2, 6, 18.