Hey guys,
First of all, its been quite a day for us over here. We've been working non-stop to bring all the search functions back to normal and we appreciate your patience and humor as we make progress. I especially like the gremlins gifs - very appropriate!
So here's whats up: No data is missing.
We are piloting a new feature (new feature X) internally that is really awesome. I cannot share details about the feature itself, but I can tell you it adds data to our core search document structure and that change started this bug.
Our search technology powers all our galleries, shops, mymodels and most other places that show lists of models. All models that are going to be a part of new feature X got an updated document in the search index, but there was an unexpected side effect that made them hidden. The backups of the index will not work with the current live website code - we want to move forward and fix the issue.
To get it working we have two paths:
1) Incrementally index each model that has been hidden from the index (ETA 5h - in progress)
- we manually triggered all models in shops to be added back as visible into search - this process takes time but it should help.
2) Full reindex all models (ETA 4.5h - in progress)
- this is our normal route to get new data into our search index, but new feature X caused it to barf RIGHT AT THE END!!!!!! TWICE!!!!!!!
- we have put extra measures into the reindexing code to prevent the last-minute barfing <GREMLIN> but we are still waiting for this process to finish.
The end result is that you guys had a soul-crushing day without your models displaying in shops, worrying that models might be missing, and generally feeling angst and showing fangs to Shapeways.
As soon as we have a final fix up and running, we'll post back to this thread.
I hope this view into the issue and this joke will help ease some of the frustation you experienced today: