Here is my example of a continuous function which maps rationals to rationals, but is not a rational function:
F(x) = (-1)^floor(x) * frac(|x|) * (frac(|x|) - 1),
where frac(x) is the fractional part of x.
--well, that was too easy. I should have said "not a piecewise rational function" to make it harder. Or perhaps, as Dan Asimov suggested, demanding it be analytic, or Cinfinity. And one can similarly ask: is there an analytic or Cinfinity function that maps algebraic numbers to algebraic numbers, but is not an algebraic function? Here's one that comes close. Consider the "Klein j-invariant" https://en.wikipedia.org/wiki/J-invariant which is sort of the most fundamental "modular function." "In 1937 Theodor Schneider proved that if X is a quadratic irrational number in the upper half plane then j(X) is an algebraic integer. In addition he proved that if X is an algebraic number but not imaginary quadratic then j(X) is transcendental." Are all algebraic integers generated in this way? Here's another interesting real valued function F(X) of real X. Let [a0; a1,a2,a3,a4...] be the continued fraction expansion of X. For each a[j] which is (i) greater than 4, and (ii) both its neighbors a[j+1] and a[j-1] also are greater than 4, replace a[j] by a[j]+1. This editing process converts X into F(X). This function F(X) has the properties that (1) if X is rational, then F(X) is too. (2) F(X) is not a rational function. Nor is it a piecewise rational function (or anyhow if it were considered one then the number of pieces would be uncountably infinite and the breakpoints would be dense everywhere). (3) if X is quadratic irrational, which happens exactly when X's continued fraction is ultimately-periodic, then F(X) also is. But F(X) is not an algebraic function. (4) I cannot believe F(X) is analytic. (5) Is F(X) continuous? (I think yes. Certainly if X and Y are both rational and |X-Y|-->0, then |F(X)-F(Y)|-->0.) (6) If F(X) differentiable? Well, it certainly is possible to define a derivative F'(X) whenever X is a rational number. Indeed, I think the Kth derivative may be defined whenever X is a rational number, for any K=1,2,3,...? I'm not sure whether these are continuous functions of X; but if they all were then F(X) would be Cinfinity. There also are interesting variants of this construction, such as, a[j] is edited to become, not a[j]+1, but rather a[j]+2^(-j).