When $X, Y$ are two topological spaces there is a commonly used topology on the set of continuous functions $X \to Y$ called the compact-open topology. Under mild hypotheses ($X$ locally compact, which is true here) this describes the exponential object $Y^X$ in the category of topological spaces and so has a very pleasant universal property, namely that there is a natural bijection
$$\text{Hom}_{\text{Top}}(X, Z^Y) \cong \text{Hom}_{\text{Top}}(X \times Y, Z).$$
For example, a hom*otopy between two continuous functions $X \to Y$ is a map $[0, 1] \to Y^X$, the evaluation map $X \times Y^X \to Y$ is continuous, etc. So for $G$ a topological group and $X$ a locally compact space we have
$$\text{Hom}_{\text{Top}}(G, X^X) \cong \text{Hom}_{\text{Top}}(G \times X, X).$$
This implies in particular that if $G$ acts faithfully on $X$ and carries the subspace topology with respect to the inclusion $G \to X^X$, then the action map $G \times X \to X$ is continuous (although it doesn't imply that $G$ is a topological group with respect to this topology!). On the other hand, your proposal only asks for continuity in $G$ for fixed $x \in X$ and doesn't seem to me to get joint continuity.
So, what topology do we get when $SO(3)$ is topologized as a subspace of the space of maps $\mathbb{R}^3 \to \mathbb{R}^3$ with the compact-open topology? We can actually prove a more general fact about the compact-open topology on the space of linear maps between two finite-dimensional real vector spaces, as follows. First, recall that a finite-dimensional real vector space has a unique topology with respect to which addition and scalar multiplication are continuous (which agrees with the product / Euclidean topology if we pick a basis and an isomorphism to some $\mathbb{R}^d$). This is also the topology induced by any norm, and below I'll call it the Euclidean topology.
Proposition: Let $V, W$ be finite-dimensional real vector spaces (with Euclidean topologies). Consider the vector space $[V, W]$ of linear maps $V \to W$, topologized as a subspace of the space $\text{Hom}_{\text{Top}}(V, W)$ of continuous maps $V \to W$, with the compact-open topology. This topology agrees with the Euclidean topology.
Proof. We'll use the fact that when $Y$ is a metric space the compact-open topology on $Y^X$ is the topology of compact convergence. Pick norms $\| \cdot \|_V, \| \cdot \|_W$ on $V, W$ (which you can take to be Euclidean if you prefer). The Euclidean topology on $[V, W]$ is induced by the operator norm
$$\| T \| = \sup_{\| v \|_V \le 1} \| T(v) \|_W.$$
We'll show that the topology of compact convergence, and hence the compact-open topology, is also the topology induced by the operator norm. The topology of compact convergence is, by definition, the topology in which a sequence $T_i$ of linear maps $V \to W$ converges iff it converges uniformly on compact subspaces of $V$. Since every compact subspace of $V$ is contained in a closed ball centered at the origin (which conversely is compact by the Heine-Borel theorem), equivalently $T_i$ converges compactly iff it converges uniformly on closed balls centered at the origin. And since linear maps are hom*ogeneous, by scaling, $T_i$ converges compactly iff it converges uniformly on the closed ball of radius $1$ centered at the origin. But this is the same as converging with respect to the operator norm. $\Box$
(This argument is actually new to me and suggests the nice intuition that the compact-open topology is like a nonlinear generalization of the operator norm, at least if $X$ is locally compact.)
Corollary: The compact-open topology on $SO(3)$, thought of as a subspace of $\text{Hom}_{\text{Top}}(\mathbb{R}^3, \mathbb{R}^3)$, agrees with the Euclidean topology.
I've come back to studying these concepts and finally understand what I was missing here. I will correct this proof for the multiplication operation.
First I will show that the map $p=\pi \times \pi: G \times G \rightarrow \left(G/\Gamma\right) \times \left(G/\Gamma\right)$ is a quotient map. It is obviously surjective and continuous, as the product of two surjective and continuous maps. (Here is what I missed previously:) The quotient map $\pi: G\rightarrow G/\Gamma$ is open (from Problem 3-15a) and so $p$ is open as the finite product of open maps (from Problem 1), and so $p$ is a quotient map.
The fibers of $p$ are of the form $$ p^{-1}\left(a\Gamma, b\Gamma\right) = a\Gamma \times b\Gamma. $$ Note that the mapping $\pi \circ \mu: G\times G \rightarrow G/\Gamma$ is constant on these fibers: if $\left(x,y\right), \left(u,v\right) \in p^{-1}\left(a\Gamma, b\Gamma\right) =a\Gamma \times b\Gamma$, then $x = \gamma u, y = v\gamma'$ for some $\gamma, \gamma'\in \Gamma$, and so $$ \pi\circ \mu\left(x,y\right) = xy\Gamma = \gamma uv \gamma' \Gamma = uv \Gamma = \pi \circ \mu\left(u,v\right) $$ since $\Gamma$ is a normal subgroup. Thus is is constant on these fibers. $\pi \circ \mu$ is continuous as the composition of continuous maps. Hence we can pass this to the quotient to obtain the continuous map $\overline{\psi}:\left(G/\Gamma\right) \times \left(G/\Gamma\right) \rightarrow G/\Gamma$ given by $\overline{\psi}\left(a\Gamma, b\Gamma\right) = ab\Gamma$ which is exactly the multiplication map on $\left(G/\Gamma\right)\times \left(G/\Gamma\right)$, hence the latter is continuous.
Thus, both operations are continuous and so $G/\Gamma$ is a topological group.
