If its reasonably difficult to decompose iterations it seems that functions could be attractive public/private keys ... of course this doesn't necessarily mean they must be represented through math symbols and what not.
Let's define f_n(x) to be the n'th iteration of f(x), that is f(f(f(f(x...)))) n times.
So, set the equation
f_n(x)=g(x)
then define the 1/n'th iterate of g(x) to be f(x)
f(x)=g_1/n(x)

Given a mathematical representation, it might be slightly easier to find the 1/n'th iterate than if I gave you a table of numbers.

For example
Find the 1/5th iterate of ln(x)
that is, find an h(x) s.t. h_5(x)=ln(x) for some interval ... for all I know, it could be impossible to represent h(x) algebraically.

An example of this comes to mind, but it isn't a finite example.  Consider the expression x*e^x=k => x=k/(e^x) | k > 0
Let f(x)=k/(e^x)
if we iterate f(x) is tends towards Lambert's W function, W(k).
We could represent the principle solution of W(k) as lim [n->infinity] f_n(x).

Now is there a way to go backwards?

A public/private key could be two functions one composed of the other a random number of times.

Consider
f(1)=4
f(2)=5
f(3)=8
f(4)=2
f(5)=7
f(6)=1
f(7)=3
f(8)=6

Let's say g(x)=f_3(x)
g(1)=5
g(2)=3
g(3)=1
g(4)=7
g(5)=8
g(6)=2
g(7)=6
g(8)=4

Now this is easy to bruteforce (since its small) if you don't know that its composed of f 3 times.  However, if you know it is 3, then you know that g_3(x)=f(x) since 9 mod 8 = 1
f and g are periodic that is f_8n(x)=x.  However, suppose g=f_6(x), how then could you find f(x) without bruteforcing?  You could note that g(x) is period 4 since 8/gcd(8,6)=4... but then what?
Suppose you are working with function map that has a domain over thousands of values, suppose we used multidimensional functions.  Not sure how you could use composed functions to encrypt data though... It's only an idea