Automatic orthographic to 3d model

I've seen plenty of 3d modeling programs that are able to take an existing 3d model and produce orthographic views.
Does there exist an algorithm that lets you classify the front, back, top, bottom, left & right of a photo or drawing that recreates a 3d representation of those orthographic views?

You could try http://www.123dapp.com/catch
You would need more than just the orthographic views.

The technical term for this is photogrammetry.

Overall, the answer is "no". There's not enough information in the orthographic views. It's a subject I've chased for 30+ years.
The core issue is that an orthographic view will not properly represent any concave surfaces.
On the other hand, a large number of models in my shop were built (by hand) from nothing more than the orthographic views.
I convert the bitmaps to vector drawings using Corel or Inkscape, then import the 3 views into (editor of my choice) and go to work extruding and intersecting.

If you take the trivial example of a football.. the 3 view would be two ovals and a circle.
If you simply extrude those and then apply a "intersection" operator, this is what you are left with.

Now, if you can obtain slices... where you have many more than just 3 views.. you might get somewhere.

I built this model using such a method, because I had the cross sections:

Adding to what Stony has said, there is one way that I know of to automatically do what he has described and that is by using Pixologic's ShadowBox command in Zbrush. It would come out looking exactly like Stony's image for a football, but it does it automatically using the three reference images. It also lets you make adjustments by drawing on the reference images and it updates whatever you draw in real time.

Then again, it doesn't take much effort to extrude and then make an intersection out of the three extrusions.

Check out 123D by Autodesk. You won't get away with using just those views, but it will convert images into a 3D model if you have enough of them of good quality