submitting a 16MB or larger STL file

Discussion in 'Shapeways API' started by mctrivia, Apr 11, 2011.

  1. mctrivia
    mctrivia New Member
    Every time I try to use the api to submit a file under 16MB it works. If I try a file over 16MB it fails and through no error codes just stops running. Anyone else having this problem?
     
  2. horst
    horst New Member
    This has got to be a glitch since it's not by design; our model size limit is much larger than this.

    Our developers are going to look into this problem right away.

    I am interested to hear if others are experiencing the problem too, to see whether it's isolated and might somehow have a different cause.

    -Ben
     
  3. hans_lambermont
    hans_lambermont Well-Known Member
    Try to use a single model file encode, (the double encode is also still supported), this backwards compatible API fix should allow larger file uploads.
     
  4. icc
    icc New Member
     
    Last edited: May 30, 2011
  5. mctrivia
    mctrivia New Member
    i am currently using:
    Code:
    $fileHandle = file_get_contents($dir . $fileName);
    
                $model = array ('title'     => $name,
                                'file'      => base64_encode($fileHandle),
                                'filename'  => $fileName, ...
    

    not sure what you would have me change. I have successfully uploaded 19MB files. but there is no errors on failure just hangs the script.

    I have written in code to auto detect script hangs but it takes up to an hour to detect the hang.
     
  6. hans_lambermont
    hans_lambermont Well-Known Member
    This :

    'file' => $fileHandle,

    The API mentions base64_encode but does not expect the user to do it. That's the tricky part and the reason why lots of ppl. including our developers do this wrong. The PHP SOAP API does the encode itself already.

    -- Hans
     
  7. mctrivia
    mctrivia New Member
    thanks I will make that change imedietly.

    Now if only I could figure out why as soon as I add a material restriction to the upload it fails. trying to add
    'materials' => array(63),

    to set to ceramic only but it fails the upload tests.
     
  8. barrys
    barrys New Member
    You should use the material name instead:

    'materials' => array('Glazed Ceramics'),

    should do the trick. Note the material name does need to be an exact match.
     
  9. mctrivia
    mctrivia New Member
    thanks will try that in about an hour. Don't know why I just assumed it would be by id not name.
     
  10. mctrivia
    mctrivia New Member
    all seems to be working so far. Here is my beta release of my first Instant Co-Create Model. I will be releasing the locks on the other fields once I have done some more testing and have successfully restricted any inputs that may cause to hang. System auto detects a crashed Damon after 1 hour and restarts but that is a long delay.

    http://icc.propmodule.com/random dot cup.html


    Now if only there was a way I could integrate the form into your page so it could be on a co-create model instead of an outside link.
     
  11. hans_lambermont
    hans_lambermont Well-Known Member
    Just tried your link, cool it works :) I've 'designed' my own mug !
     
  12. mctrivia
    mctrivia New Member
    thanks. I have some even cooler designs I plan to get up this way soon.

    At present I have to use curl to poll shapeways to find out when you are done processing the model(step 3 processing). Would be nice to be able to do this through API and even be able to get rendered image through API.

    Ideally instead of polling I could submit a callback URL so you can tell me when it is done. This would drop bandwidth a air bit as presently I am polling your site every 5 seconds until the processing is done.
     
  13. hans_lambermont
    hans_lambermont Well-Known Member
    Interesting idea. What do you propose as fallback when this callback URL connection attempt fails ?
     
  14. mctrivia
    mctrivia New Member
    i would try twice then give up. If 2 attempts fail then my server has likely crashed(happened 3 times in 3 years). I can always retry polling if it has been more then 15min.

    for record keeping it would be nice if the callback url would be called for the following

    processing complete
    model sold
    model shipped
    model order canceled do to not printable


    i could pass you a url such as http://icc.propmodule.com/callback/$id/$error and you could replace $... with apropriate info

    This way server can handle things automatically instead of sending me a tone of emails and me having to write a pop3 server to handle them
     
  15. woody64
    woody64 Well-Known Member
    Yes, my current implementtaion shows the same.
    Smaller files are submitted, larger ones stop without a further remark:
    I get this:
    string(20) "SoapAPI::wsdlconnect" string(28) "Connection successfully made" string(14) "SoapAPI::login" string(57) "we got the following session : 1ujpr06jvduamm6tk1e6e6oc05" string(10) "uploadFile"

    Additionally I run in memory problems of my php installation. It's hosted on a provider and reports memory allocation failures wih larger files.

    Woody64
     
  16. mctrivia
    mctrivia New Member
    I use 512MB of RAM for my render script and 256MB for upload to make sure it is always shapeways causing the failure. It sucks that shapeways abandend the API before finishing it or fixing bugs like this. If you use binary STL makes it less likely to hit the limit but it is still really easy to hit.
     
  17. woody64
    woody64 Well-Known Member
    I've already switched to binary stl but I'm running also very quickly into the limit.
    Joining more then 5 files and I'm in.

    At the end there's also the possibility to zip but that's oncemore an extra step/workaround.

    But at least the concept is proven, which makes it also possible to upload per hand.

    Woody64
     
    Last edited: Oct 9, 2011
  18. mctrivia
    mctrivia New Member
    i need to start experimenting with zipping on the fly. Some experimenting shows that zipped stl files are about 50% original size. At that compression ratio I am getting a lot closer to the 1,000,000 triangle limit.
     
  19. abite
    abite New Member
    Hi mctrivia.

    Is everything OK for submitting files now?
    I have the same problem. I had thought my case was exactly same as yours, but I figured out the size of file didn't matter in my case.
    If the file size is somewhat large and it takes over 3 minutes(180 seconds), it fails receiving any result or error. However, the file is uploaded. There is no errors on failure just hangs the script.... same as your case.
    Less than 3 minutes, everything is perfact.

    As hans mentioned, I did change the code like below.
    'file' => $fileHandle
    but, still in the same situation.

    How did you solve the problem? Did you just do what hans said?
    I have tried to solve this problem for several days... checked my server, script, my computer...

    Any advice would be deeply appreciated. Thanks.

    JC
     
    Last edited: Mar 28, 2012
  20. hans_lambermont
    hans_lambermont Well-Known Member
    Hi all, reproduced the issues you showed while the upload itself claims success though was slower than I had expected : it took 18 minutes to upload an STL of 49 MiB. I uploaded from a place with 1.4 Mbit/s max uplink speed, so the file should have been transferred in 5 minutes instead.

    The file arrived broken; only 18 bytes :-/ We'll look into that.

    -- Hans
     
    Last edited: Mar 28, 2012