Google Colaboratory is the best tool for machine learning engineer

python
Google Colaboratory is the best tool for machine learning engineer
Page content

For engineer who use Jupyter Notebook in often, web IDE named Colaboratory is the best tool. Try Colaboratory.

What is Colaboratory?

Colaboratory is a web IDE based on Jupyter notebook and hosted by Google.

Not only editor, but also runtime is provided by Google, users don’t need to create the environment for development.

Everyone can use Colaboratory without Google account, however, links the account, users earn lots of advantages.

colaboratory_top

The good points of Colaboratory

Free of charges

About usage fee is written in the FAQ.

Is it free to use?

Yes. Colaboratory is a research project that is free to use.

Free to use. Awesome!! In addition, running code with GPU is also free to use.

GPU is available

Users can select GPU as code runtime in Colaboratory.

In case of execution of high cost operation, using a local machine, the fan rotates with full power and other operations of the machine is delayed, but Colaboratory uses computer resources on the cloud.

Change settings is just 2 steps.

  • Select [Runtime] > [Change runtime type] from Menu
  • Change “Hardware accelerator” to “GPU” and [Save]

Now check running on GPU with executing following codes.

1import tensorflow as tf
2tf.test.gpu_device_name()

The device name is printed, in case of running on CPU, empty string is.

switch_to_gpu_3

TPU is available

In September 2018, Colaboratory supports TPU（Tensor Processing Unit） runntime.

Google sells Edge TPU for edge cmomputing, developer can execute codes for TPU.

Collaborate with Google Drive

Just Google’s service, easy to collaborate with Google Drive. Google Drive supports Jupyter notebook file saving with .ipynb extension.

save_to_drive

In order to upload an .ipynb file existed on the local machine, select [Upload notebook] and drag and drop to the upload form.

upload_notebook

When .ipynb files is opened on the Google Drive with [Colaboratory] (https://colab.research.google.com/), display popup. And file icons looks different in file created by Colaboratory or in uploaded.

Default code snipets

Colaboratory enhances Jupyter notebook, supports code snipets. Select the code snipets and paste it to cell of Jupyter notebook, particularly, code snipets for Google Drive API is useful.

use_snipet

Command/Ctrl + Alt + P is a keyboard shortcut for opening snipet window.

Choices runtime machine

This is unusual use case, switching machine runtime from Colaboratory to local machine. The setting is somewhat special, so summarize below.

Setup local machine

  • Install Jupyter noteboook
1pip install jupyter\[notebook\]
  • Enable serverextension
1pip install jupyter_http_over_ws
2jupyter serverextension enable --py jupyter_http_over_ws
  • Allow access from Colaboratory
1jupyter notebook --NotebookApp.allow_origin='https://colab.research.google.com' --port=8888

Setting in Colaboratory

Select “Connect” menu on the upper right side of the screen, and select “Connect to local runtime…”, and then, code is run on local machine.

switch_runtime

Important notice

Colaboratory is not suitable for running for a long time

Colaboratory is grateful to be able to use GPU, but Google doesn’t recommend running the runtime for a long time, especially not to use for crypt currency mining.

It’s written in the FAQ.

How may I use GPUs and why are they sometimes unavailable?

Colaboratory is intended for interactive use. Long-running background computations, particularly on GPUs, may be stopped. Please do not use Colaboratory for cryptocurrency mining. Doing so is unsupported and may result in service unavailability. We encourage users who wish to run continuous or long-running computations through Colaboratory’s UI to use a local runtime.

I found a developer who tried to run GPU for more than 12 hours, but Google doesn’t announce about limit of continuous execution time per execution.

Need to check security policy for executing code on cloud

In companies with strict security governance, Colaboratory may not be available because of security policy. Even if temporary use, valuable assets in company is uploaded on the cloud.

Be careful for python module differences when swithcing runtime

When swithcing runtime from Colaboratory to local machine, Python modules installed in Colaboratory are different from in local machine.

Execute Python code as follows, check installed modules.

1import pkg_resources
2[pkg for pkg in pkg_resources.working_set]

Colaboratory has modules as follows in default.

  1[xgboost 0.7.post4 (/usr/local/lib/python3.6/dist-packages),
  2 wheel 0.31.0 (/usr/local/lib/python3.6/dist-packages),
  3 Werkzeug 0.14.1 (/usr/local/lib/python3.6/dist-packages),
  4 webencodings 0.5.1 (/usr/local/lib/python3.6/dist-packages),
  5 wcwidth 0.1.7 (/usr/local/lib/python3.6/dist-packages),
  6 urllib3 1.22 (/usr/local/lib/python3.6/dist-packages),
  7 uritemplate 3.0.0 (/usr/local/lib/python3.6/dist-packages),
  8 traitlets 4.3.2 (/usr/local/lib/python3.6/dist-packages),
  9 tornado 4.5.3 (/usr/local/lib/python3.6/dist-packages),
 10 testpath 0.3.1 (/usr/local/lib/python3.6/dist-packages),
 11 terminado 0.8.1 (/usr/local/lib/python3.6/dist-packages),
 12 termcolor 1.1.0 (/usr/local/lib/python3.6/dist-packages),
 13 tensorflow 1.7.0 (/usr/local/lib/python3.6/dist-packages),
 14 tensorboard 1.7.0 (/usr/local/lib/python3.6/dist-packages),
 15 sympy 1.1.1 (/usr/local/lib/python3.6/dist-packages),
 16 statsmodels 0.8.0 (/usr/local/lib/python3.6/dist-packages),
 17 six 1.11.0 (/usr/local/lib/python3.6/dist-packages),
 18 simplegeneric 0.8.1 (/usr/local/lib/python3.6/dist-packages),
 19 setuptools 39.1.0 (/usr/local/lib/python3.6/dist-packages),
 20 seaborn 0.7.1 (/usr/local/lib/python3.6/dist-packages),
 21 scipy 0.19.1 (/usr/local/lib/python3.6/dist-packages),
 22 scikit-learn 0.19.1 (/usr/local/lib/python3.6/dist-packages),
 23 scikit-image 0.13.1 (/usr/local/lib/python3.6/dist-packages),
 24 rsa 3.4.2 (/usr/local/lib/python3.6/dist-packages),
 25 requests 2.18.4 (/usr/local/lib/python3.6/dist-packages),
 26 requests-oauthlib 0.8.0 (/usr/local/lib/python3.6/dist-packages),
 27 pyzmq 16.0.4 (/usr/local/lib/python3.6/dist-packages),
 28 PyYAML 3.12 (/usr/local/lib/python3.6/dist-packages),
 29 PyWavelets 0.5.2 (/usr/local/lib/python3.6/dist-packages),
 30 pytz 2018.4 (/usr/local/lib/python3.6/dist-packages),
 31 python-dateutil 2.5.3 (/usr/local/lib/python3.6/dist-packages),
 32 pyparsing 2.2.0 (/usr/local/lib/python3.6/dist-packages),
 33 Pygments 2.1.3 (/usr/local/lib/python3.6/dist-packages),
 34 pyasn1 0.4.2 (/usr/local/lib/python3.6/dist-packages),
 35 pyasn1-modules 0.2.1 (/usr/local/lib/python3.6/dist-packages),
 36 ptyprocess 0.5.2 (/usr/local/lib/python3.6/dist-packages),
 37 psutil 5.4.5 (/usr/local/lib/python3.6/dist-packages),
 38 protobuf 3.5.2.post1 (/usr/local/lib/python3.6/dist-packages),
 39 prompt-toolkit 1.0.15 (/usr/local/lib/python3.6/dist-packages),
 40 portpicker 1.2.0 (/usr/local/lib/python3.6/dist-packages),
 41 plotly 1.12.12 (/usr/local/lib/python3.6/dist-packages),
 42 pip 10.0.1 (/usr/local/lib/python3.6/dist-packages),
 43 Pillow 4.0.0 (/usr/local/lib/python3.6/dist-packages),
 44 pickleshare 0.7.4 (/usr/local/lib/python3.6/dist-packages),
 45 pexpect 4.5.0 (/usr/local/lib/python3.6/dist-packages),
 46 patsy 0.5.0 (/usr/local/lib/python3.6/dist-packages),
 47 pandocfilters 1.4.2 (/usr/local/lib/python3.6/dist-packages),
 48 pandas 0.22.0 (/usr/local/lib/python3.6/dist-packages),
 49 pandas-gbq 0.4.1 (/usr/local/lib/python3.6/dist-packages),
 50 opencv-python 3.4.0.12 (/usr/local/lib/python3.6/dist-packages),
 51 olefile 0.45.1 (/usr/local/lib/python3.6/dist-packages),
 52 oauthlib 2.0.7 (/usr/local/lib/python3.6/dist-packages),
 53 oauth2client 4.1.2 (/usr/local/lib/python3.6/dist-packages),
 54 numpy 1.14.3 (/usr/local/lib/python3.6/dist-packages),
 55 notebook 5.2.2 (/usr/local/lib/python3.6/dist-packages),
 56 nltk 3.2.5 (/usr/local/lib/python3.6/dist-packages),
 57 networkx 2.1 (/usr/local/lib/python3.6/dist-packages),
 58 nbformat 4.4.0 (/usr/local/lib/python3.6/dist-packages),
 59 nbconvert 5.3.1 (/usr/local/lib/python3.6/dist-packages),
 60 mpmath 1.0.0 (/usr/local/lib/python3.6/dist-packages),
 61 mistune 0.8.3 (/usr/local/lib/python3.6/dist-packages),
 62 matplotlib 2.1.2 (/usr/local/lib/python3.6/dist-packages),
 63 MarkupSafe 1.0 (/usr/local/lib/python3.6/dist-packages),
 64 Markdown 2.6.11 (/usr/local/lib/python3.6/dist-packages),
 65 Keras 2.1.6 (/usr/local/lib/python3.6/dist-packages),
 66 jupyter-core 4.4.0 (/usr/local/lib/python3.6/dist-packages),
 67 jupyter-client 5.2.3 (/usr/local/lib/python3.6/dist-packages),
 68 jsonschema 2.6.0 (/usr/local/lib/python3.6/dist-packages),
 69 Jinja2 2.10 (/usr/local/lib/python3.6/dist-packages),
 70 ipython 5.5.0 (/usr/local/lib/python3.6/dist-packages),
 71 ipython-genutils 0.2.0 (/usr/local/lib/python3.6/dist-packages),
 72 ipykernel 4.6.1 (/usr/local/lib/python3.6/dist-packages),
 73 idna 2.6 (/usr/local/lib/python3.6/dist-packages),
 74 httplib2 0.11.3 (/usr/local/lib/python3.6/dist-packages),
 75 html5lib 0.9999999 (/usr/local/lib/python3.6/dist-packages),
 76 h5py 2.7.1 (/usr/local/lib/python3.6/dist-packages),
 77 grpcio 1.11.0 (/usr/local/lib/python3.6/dist-packages),
 78 googleapis-common-protos 1.5.3 (/usr/local/lib/python3.6/dist-packages),
 79 google-resumable-media 0.3.1 (/usr/local/lib/python3.6/dist-packages),
 80 google-colab 0.0.1a1 (/usr/local/lib/python3.6/dist-packages),
 81 google-cloud-translate 1.3.1 (/usr/local/lib/python3.6/dist-packages),
 82 google-cloud-storage 1.8.0 (/usr/local/lib/python3.6/dist-packages),
 83 google-cloud-language 1.0.1 (/usr/local/lib/python3.6/dist-packages),
 84 google-cloud-core 0.28.1 (/usr/local/lib/python3.6/dist-packages),
 85 google-cloud-bigquery 1.1.0 (/usr/local/lib/python3.6/dist-packages),
 86 google-auth 1.4.1 (/usr/local/lib/python3.6/dist-packages),
 87 google-auth-oauthlib 0.2.0 (/usr/local/lib/python3.6/dist-packages),
 88 google-auth-httplib2 0.0.3 (/usr/local/lib/python3.6/dist-packages),
 89 google-api-python-client 1.6.7 (/usr/local/lib/python3.6/dist-packages),
 90 google-api-core 1.1.1 (/usr/local/lib/python3.6/dist-packages),
 91 gast 0.2.0 (/usr/local/lib/python3.6/dist-packages),
 92 future 0.16.0 (/usr/local/lib/python3.6/dist-packages),
 93 entrypoints 0.2.3 (/usr/local/lib/python3.6/dist-packages),
 94 decorator 4.3.0 (/usr/local/lib/python3.6/dist-packages),
 95 cycler 0.10.0 (/usr/local/lib/python3.6/dist-packages),
 96 crcmod 1.7 (/usr/local/lib/python3.6/dist-packages),
 97 chardet 3.0.4 (/usr/local/lib/python3.6/dist-packages),
 98 certifi 2018.4.16 (/usr/local/lib/python3.6/dist-packages),
 99 cachetools 2.0.1 (/usr/local/lib/python3.6/dist-packages),
100 bleach 1.5.0 (/usr/local/lib/python3.6/dist-packages),
101 beautifulsoup4 4.6.0 (/usr/local/lib/python3.6/dist-packages),
102 astor 0.6.2 (/usr/local/lib/python3.6/dist-packages),
103 absl-py 0.2.0 (/usr/local/lib/python3.6/dist-packages)]

Jupyter notebook has modules as follows in default.

 1[widgetsnbextension 3.2.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 2 webencodings 0.5.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 3 wcwidth 0.1.7 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 4 traitlets 4.3.2 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 5 tornado 5.0.2 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 6 testpath 0.3.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 7 terminado 0.8.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 8 six 1.11.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
 9 simplegeneric 0.8.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
10 setuptools 28.8.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
11 Send2Trash 1.5.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
12 qtconsole 4.3.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
13 pyzmq 17.0.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
14 python-dateutil 2.7.2 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
15 Pygments 2.2.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
16 ptyprocess 0.5.2 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
17 prompt-toolkit 1.0.15 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
18 pip 10.0.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
19 pickleshare 0.7.4 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
20 pexpect 4.5.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
21 parso 0.2.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
22 pandocfilters 1.4.2 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
23 notebook 5.4.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
24 nbformat 4.4.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
25 nbconvert 5.3.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
26 mistune 0.8.3 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
27 MarkupSafe 1.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
28 jupyter 1.0.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
29 jupyter-core 4.4.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
30 jupyter-console 5.2.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
31 jupyter-client 5.2.3 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
32 jsonschema 2.6.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
33 Jinja2 2.10 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
34 jedi 0.12.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
35 ipywidgets 7.2.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
36 ipython 6.3.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
37 ipython-genutils 0.2.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
38 ipykernel 4.8.2 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
39 html5lib 1.0.1 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
40 entrypoints 0.2.3 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
41 decorator 4.3.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
42 bleach 2.1.3 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
43 backcall 0.1.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages),
44 appnope 0.1.0 (/Users/soudegesu/.pyenv/versions/3.6.1/envs/test/lib/python3.6/site-packages)]

Colaboratory includes Machine learning, scientific computing, and Google API libraries etc. If necessary, install the same libraries to the local machine.

Conclusion

Google Colaboratory is the best tool for machine learning engineer, because …

  • Free to use any processor(CPU, GPU, and TPU)
  • Easy to collaborate with Google services
  • No preparation to write and execute python code