Official solution
As, \[\cos B = {3 \over 5} \Rightarrow B = 53^\circ \]
As, \[R = 5 \Rightarrow {c \over {\sin c}} = 2R\]
\[ \Rightarrow {5 \over {10}} = \sin c \Rightarrow C = 30^\circ \]
Now, \[{b \over {\sin B}} = 2R \Rightarrow b = 2(5)\left( {{4 \over 5}} \right) = 8\]
Now, by cosine formula
\[\cos B = {{{a^2} + {c^2} - {b^2}} \over {2ac}}\]
\[ \Rightarrow {3 \over 5} = {{{a^2} + 25 - 64} \over {2(5)a}}\]
\[ \Rightarrow {a^2} - 6a - 3g = 0\]
\[\therefore\] \[a = {{6 \pm \sqrt {192} } \over 2} = {{6 \pm 8\sqrt 3 } \over 2}\]
\[ \Rightarrow 3 + 4\sqrt 3 \] (Reject \[a = 3 - 4\sqrt 3 \])
Now, \[\Delta = {{abc} \over {4R}} = {{(3 + 4\sqrt 3 )(8)(5)} \over {4(5)}} = 2(3 + 4\sqrt 3 )\]
\[ \Rightarrow \Delta = (6 + 8\sqrt 3 )\]
\[\Rightarrow\] Option (3) is correct.