Learn how to use the Elastic App Search APIs in order to automatically move engines from Swiftype App Search to Elastic Cloud App Search.

If you are a current App Search user on Swiftype.com and you’d like to move to Elastic Cloud to benefit from the latest developments — look no further! Here’s a sampling of all the long term benefits you’ll enjoy after migrating:<ul><li aria-level="1">Faster performance, plus data locality</li><li aria-level="1">More flexibility and scalability&nbsp;</li><li aria-level="1">Easier log and analytics management</li></ul><table style="background: #FFFFD2;"><tbody><tr><td>Note: If you’re not yet sure about moving you can read about the <a href="https://www.elastic.co/blog/migrating-from-swiftype-app-search-to-elastic-cloud">benefits of migrating</a>. </td></tr></tbody></table>In this blog we look at automating engine transfers from App Search on Swiftype.com to App Search on Elastic Cloud, using nothing but simple Python scripts and the available APIs. At the end of this process, you’ll be left with exactly the same engines on Elastic Cloud that you had on Swiftype.com.&nbsp;Before proceeding, we highly recommend looking over the <a href="/guide/en/enterprise-search/current/migrating-from-app-search-on-swiftype-com.html">migration guide</a> to get an understanding of the steps involved, as well as the impact of this migration.You’ll need your existing App Search engines on Swiftype.com and a new deployment on Elastic Cloud with Enterprise Search enabled to get ready to move. If you’re not sure how to get started, have a look at this <a href="/training/app-search-quick-start">Quick Start</a> to get things up and running in Elastic Cloud.<h2>Scripting the steps</h2>Let’s consider the steps we must take along with code examples that allow us to automate things via some Python scripting (see the full code at the end of the blog post).<h3>Step 1: Download the Swiftype App Search engine metadata</h3><ul><li aria-level="1">First, we use App Search APIs to download all the engine metadata we need.</li><li aria-level="1">To script this step, we used 1_get_settings_st.py<ul><li aria-level="2">We’ll need to input the engine names, a local directory to store files temporarily, and the API key/endpoint for Swiftype App Search.</li></ul></li><li aria-level="1">Then save and run the script. This will download the schema, settings, curations, and synonyms for each engine locally to disk in .json files.</li></ul><h3>Step 2: Create new Elastic Cloud App Search engines</h3><ul><li aria-level="1">For this step, 2_create_eng_ec iterates the engine names and creates them on Elastic Cloud.<ul><li aria-level="2">We’ll need to input the same engine names and directory from the previous step.</li><li aria-level="2">This time we'll&nbsp;need to input the Elastic Cloud App Search API key/endpoint.</li></ul></li><li aria-level="1">It also populates the schema for each of the created engines with the previously downloaded schemas.</li></ul><h3>Step 3: Backfill documents in Elastic Cloud App Search (optional)</h3><ul><li aria-level="1">We strongly recommend performing this step by backfilling directly from source. It alleviates unnecessary complications. However, we know this isn’t always possible, so the script will move these over for you if needed.</li><li aria-level="1">The script 3_migrate_docs iterates pages of documents in Swiftype App Search and copies them over to App Search on Elastic Cloud.<ul><li aria-level="2">Once again, we’ll need the API key/endpoint for both instances of App Search.</li><li aria-level="2">This time we don’t use any local storage — simply hold documents in memory to be migrated over in chunks.</li></ul></li><table style="background-position: initial;background-color: rgb(255, 255, 210);font-size: 14px;width: 789.516px;"><tbody><tr><td>Note:&nbsp;There is a cap of 10,000 documents that can be retrieved via the List Documents API so if the number of documents in the engine is above this number, the script will not move all your documents. Instead, you’ll need to use the Search API to retrieve fewer than 10,000 documents and populate them in chunks. We don’t have a standard piece of code to automate this process as it will be unique to the data being moved - however, if you’d like assistance with this, Elastic offers <a href="https://www.elastic.co/consulting">Consulting</a> and Support services to help you through it.</td></tr></tbody></table></ul><h3>Step 4: Migrate the search settings, curations, and synonyms</h3><ul><li aria-level="1">These were previously downloaded in Step 1.</li><li aria-level="1">The script 4_add_settings.py will iterate all engines and add all of the settings for these engines in Elastic Cloud App Search.<ul><li aria-level="2">We’ll need to populate the App Search API key/endpoint and local directory path again.</li></ul></li></ul><h3>Step 5: Migrate the result settings and user roles</h3><ul><li aria-level="1">This is a manual step. We will need to migrate the <a href="/guide/en/app-search/current/result-settings-guide.html">result settings</a>&nbsp;and <a href="/guide/en/app-search/current/role-based-access-control-guide.html">user roles</a>&nbsp;from Swiftype to Elastic Cloud.</li></ul><h2>Finishing up</h2>After testing, running, and verifying the results of the steps above, there are a few things to iron out:<ul><li aria-level="1">The client applications currently consuming App Search APIs on Swiftype.com will need to be switched over to Elastic Cloud.</li><li aria-level="1">The parity between both engines should be tested by sending queries and writes to the new engines.<ul><li aria-level="2">If documents were added to the old engine while the migration was in progress, the new data needs to be copied over to ensure it’s completely up to date.</li></ul></li><li aria-level="1">The <a href="/guide/en/app-search/current/analytics.html">analytics</a> from Swiftype.com can be exported and backed up if you'll need them.</li><li aria-level="1">Finally, after verifying that there are no new events in the Swiftype App Search API logs, we can decommission the old engines.</li></ul><h2>More info</h2>To help you learn more, we put together a <a href="/webinars/migrating-from-swiftype-app-search-to-elastic-cloud">webinar</a> that highlights all the benefits of migrating to App Search on Elastic Cloud, including an overview of the process.If you’d like to take a spin around App Search on Elastic Cloud, you can always start up a <a href="http://cloud.elastic.co/registration">free trial</a>.<h2>Appendix</h2><pre class="prettyprint">1_get_settings_st.py import requests import json # Populate API key, host, temp folder and engine names headers={'Content-Type': 'application/json','Authorization': 'Bearer private-&lt;API_KEY&gt;'} host='https://host-abc123.api.swiftype.com' data_folder='/tmp/' engines = ['dev-products','prod-products'] for engine in engines: st_api = host+'/api/as/v1/engines/'+engine+'/' print('Downloading engine metadata: '+engine) # Get Schema st_schema = st_api+'schema' r = requests.get(st_schema,headers=headers) with open(data_folder+engine+'_schema.json', "w") as file: file.write(json.dumps(r.json())) # Get search settings st_settings = st_api+'search_settings' r = requests.get(st_settings,headers=headers) with open(data_folder+engine+'_settings.json', "w") as file: file.write(json.dumps(r.json())) # Get curations st_curations = st_api+'curations' r = requests.get(st_curations,headers=headers) with open(data_folder+engine+'_curations.json', "w") as file: file.write(json.dumps(r.json())) # Get synonyms st_synonyms = st_api+'synonyms' r = requests.get(st_synonyms,headers=headers) with open(data_folder+engine+'_synonyms.json', "w") as file: file.write(json.dumps(r.json())) </pre><pre class="prettyprint">2_create_eng_ec import requests import json # Populate API key, host, temp folder and engine names headers={'Content-Type': 'application/json','Authorization': 'Bearer private-&lt;API_KEY&gt;'} data_folder='/tmp/' host='https://abc123.ent-search.westeurope.azure.elastic-cloud.com' engines = ['dev-products','prod-products'] for engine in engines: ec_api = host+'/api/as/v1/engines/' # Create engine r = requests.post(ec_api,headers=headers,json={"name": engine}) print('Create '+engine+' response: '+str(r.status_code)) # Update schema ec_schema = ec_api+engine+'/schema' schema=open(data_folder+engine+'_schema.json', 'rb').read() r = requests.post(ec_schema,headers=headers,data=schema) print('Add schema '+engine+' response: '+str(r.status_code)) </pre><pre class="prettyprint">3_migrate_docs import requests import json from sys import getsizeof # Populate API keys, hosts, engine names headers_st={'Content-Type': 'application/json','Authorization': 'Bearer private-&lt;API_KEY&gt;'} headers_ec={'Content-Type': 'application/json','Authorization': 'Bearer private-&lt;API_KEY&gt;'} st_host='https://host-abc123.api.swiftype.com' ec_host='https://abc123.ent-search.westeurope.azure.elastic-cloud.com' engines = ['dev-products','prod-products'] # Migrate documents between services, up to 10,000 for engine in engines: st_docs = st_host+'/api/as/v1/engines/'+engine+'/documents/list' ec_docs = ec_host+'/api/as/v1/engines/'+engine+'/documents' page = 0 print('Migrating docs in '+engine) while True: r_st = requests.get(st_docs,headers=headers_st,params={'page[current]':page}) docs = json.dumps(r_st.json()['results']) r_ec = requests.post(ec_docs,headers=headers_ec,data=docs) print(r_ec.status_code) # Print info, check if we break print('Page '+str(r_st.json()['meta']['page']['current'])+' of '+str(r_st.json()['meta']['page']['total_pages'])) if r_st.json()['meta']['page']['current'] == r_st.json()['meta']['page']['total_pages']: break else: page+=1 </pre><pre class="prettyprint">4_add_settings.py import requests import json # Populate API key, host, temp folder and engine names headers={'Content-Type': 'application/json','Authorization': 'Bearer private-&lt;API_KEY&gt;'} data_folder='/tmp/' host='https://abc123.ent-search.westeurope.azure.elastic-cloud.com' engines = ['dev-products','prod-products'] # Update search settings, curations and synonyms for these engines for engine in engines: ec_api = host+'/api/as/v1/engines/'+engine # Update settings ec_settings = ec_api+'/search_settings' settings=open(data_folder+engine+'_settings.json', 'rb').read() r = requests.put(ec_settings,headers=headers,data=settings) print('Adding settings for '+engine+': '+str(r.status_code)) # Update curations ec_curations = ec_api+'/curations' with open(data_folder+engine+'_curations.json') as json_file: data = json.load(json_file) for c in data['results']: del c['id'] r = requests.post(ec_curations,headers=headers,data=str(c)) print('Adding curation '+str(c)+': '+str(r.status_code)) # Update synonyms ec_synonyms = ec_api+'/synonyms' with open(data_folder+engine+'_synonyms.json') as json_file: data = json.load(json_file) for c in data['results']: del c['id'] r = requests.post(ec_synonyms,headers=headers,data=str(c)) print('Adding synonym '+str(c)+': '+str(r.status_code)) </pre>

blog-banner-search-results-package-pink.png

blog-thumb-search-results-package-pink.png

How to migrate from Swiftype App Search to Elastic Cloud

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Rathin Sawhney

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、上記で提供された連絡先に<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">特定の通知</a>が送信されることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシーポリシー</a>をご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。このフォームを送信することにより、Elasticが私の連絡先情報をGoogle Cloudに共有することに同意します。また、Elasticが私に食材パッケージを発送する目的で私の情報をベンダーに提供することにも同意します。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

送信することにより、Elasticの<a href="/jp/legal/serverless-preview-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

送信することにより、<a href="/jp/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud利用規約</a>に同意したものとみなされます。個人情報は<a href="/jp/legal/privacy-statement">Elasticのプライバシーポリシー</a>に基づいて取り扱われます。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

送信することにより、Elasticの<a href="/jp/legal/terms-of-use" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

このページのコンテンツは、選択された言語ではご利用いただけません。Elasticでは現在、コンテンツをさまざまな言語でご利用いただけるよう取り組んでいます。お使いの言語が準備されるまでお待ちくださるようお願いいたします。

Author

Articles by

Watch now (no PT)

Watch now

See all top stories

すべてのトップストーリーを見る

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Search Integrations

All Solutions

Video for

See when this webinar starts in my time zone

現在のタイムゾーンでこのウェビナー開始時間を見る

Register to Watch

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

他の予定と重なってしまった…という場合も、問題ありません。参加登録されると、録画配信をご利用いただけます。関連コンテンツのご案内もメールでお届けします。

Author

Articles by Rathin Sawhney

Elastic

How to migrate from Swiftype App Search to Elastic Cloud

Sign up for Elastic Cloud free trial