12. März 2017 // von Stefan A.

Project Babelfish

A lot has changed since Thomann started out as a small local retailer based in Treppendorf in 1954. Still based in Treppendorf, we now ship musical equipment worth over 600 million Euros a year to customers all over the world. Having reached a certain size that exceeds Germany’ need, our international customers are increasingly important for company growth. We have a dedicated shop for 18 european countries and an international shop for customers all around the world.

Our customers are what makes Thomann great and we are especially thankful for almost one million customer reviews, verified by purchase, providing valuable information to us and other music lovers. Unfortunately, because most evaluations are written in German or English (our main shop languages) a large portion are inaccessible to our international customers. Since the Babelfish is not yet discovered, and our translation team would be a bit overwhelmed with all this content, we looked for other solutions. We heard AI is all the rage now, so let computers do the job!

What’s the best online translation tool?

The contenders are Google Cloud Translate, Microsoft Translator Text API and Yandex Translate. Google also offers a “Premium” service, currently in beta, which we also included in our tests. In order to find out the best online translation service we ran some random customer reviews through the services and compared the results. This is obviously not the most scientific method but qualitative differences were found nonetheless ;-).

What is special about our use case?

  • User reviews are written in common language, with idioms and words usually not used in published books or professionally written articles.
  • Sometimes, user reviews contain grammatical errors or incomplete sentences even in their native language--usually glossed over by the reader but machines will have a harder time ignoring these
  • User reviews are context specific (on some special musical equipment), which is obvious to the human reader but not obvious to a machine
  • User reviews sometimes contain spelling mistakes which should be corrected
  • Special musician-related vocabulary

What qualities do we expect from a translation service?

  • Above all, of course the best possible machine translation
  • However, a bad literal translation is sometimes better than a grammatically perfect sentence if the semantics are changed.
  • Ease of use for programmatic implementation
  • Continuous improvement in quality
  • Acceptable response times (<2s)
  • reasonable pricing

Fight!

Service Mit dem Lambda Interface von Lexicon habe ich wohl eine gute Entscheidung getroffen. Quality
Yandex With the Lambda Interface of Lexicon, I've made a good decision. 4/5
Google With the Lambda Interface from Lexicon, I have probably made a good decision. 5/5
Microsoft With the lambda, I have probably made a good decision interface of lexicon. 3/5
Google Premium With the Lambda Interface from Lexicon, I have probably made a good decision. 5/5

Microsoft unfortunately messes up the word order, Yandex misses a word. Google flawless.

A more difficult Sentence:

Service Obwohl ich hauptsächlich Gitarre spiele, haben mich Holzblasinstrumente schon immer interessiert. Quality
Yandex Although I play mainly the guitar, have me wooden wind instruments have always been interested in. 2/5
Google Although I mainly play guitar, woodwind instruments have always interested me. 5/5
Microsoft Even though I'm playing mainly guitar, woodwinds have always interested me 4/5
Google Premium Although I mainly play guitar, woodwind instruments have always interested me. 5/5

Again Google takes the first place, Yandex has difficulties. Bing’s translation is acceptable.

A really difficult sentence:

Service Die blökenden, quitschenden und jämmerlich wimmernden Geräusche, die ich anfangs hervorbrachte, fand ich selbst einfach urkomisch. :) Und dabei hab ich nur versucht, mir den Tonumfang lt. Grifftabelle zu erarbeiten... Quality
Yandex The bleating, squeaky and pathetic whimpering noises, which I brought forth in the beginning, I found simply hilarious. :) And I've tried just, to me, the range of lt. To develop, handle table... 2/5
Google The bleating, squeaking and miserable Whimpering sounds, which I initially gave, I found myself simply hilarious. :) And I've only tried to me the range of tone according to Grifftabelle to work out ... 3/5
Microsoft I found the blökenden, enjoyed and pathetic whimpering noises that I initially brought forth, simply hilarious. :) And while I tried only to prepare me the tonal range according to the chord... 3/5
Google Premium The bleating, squeaking and miserable Whimpering sounds, which I initially gave, I found myself simply hilarious. :) And I've only tried to me the range of tone according to Grifftabelle to work out … 3/5

No one shines here, but you kind of get the sense of it.

Results

This, of course, is only a sample of our extensive testing. However it demonstrates the current state of the art in translation technology. Yandex appears to be behind its competitors. This may be due to the fact that it is more Russian focussed than the other services. Google and Microsoft are neck to neck, with a slight edge for Google.

With Google and Google Premium we found only occasional difference, sometimes even regressions to the normal translation service. In some cases, though, quality appears to be much better.

Machine translation has come a long way since its beginnings. Quality is still not comparable to an average human translator, but combined with some common sense we hope that our customers will be able to get new insights into our products they might have missed otherwise.

As for API-Usability, all services are well documented and easy to use. Microsoft and Yandex present their results in xml; Google in json. We prefer json as it is usually less cluttered. Response times were within limits for all services. All services require an attribution.

Pricing is in the range of $10 to $20 per 1 million character with Google being the most expensive, yandex in the middle and Microsoft the cheapest.

All things considered, we chose Google Premium as it provides, in our view, the best quality for our main languages.

So… how can i test it?

If you just read this article, you probably understand English. In our main stores, we usually have enough English and German reviews without the need of translating other languages. However, if you want to compare, say, your finnish skills with that of google, go ahead and try it out!

Verwandte Blogposts