Pages

Tuesday, February 25, 2014

Building a Python/Django Development Virtual Machine

Recently for a project, I needed to build a Python, Django, PostgreSQL, NGINX Development Virtual Machine(VM). Below are the steps that I followed to build this VM in VMware Fusion on MacbookPro (MBP). This post is as much about sharing my build experience as documenting the steps for my future use and potential automation.

The guidance for this procedure came from How To Install and Configure Django with Postgres, Nginx, and Gunicorn.

Installation

Ubuntu Server OS

The steps were very similar to the ones I covered in my prior post OpenStack: Quick Install using DevStack.

Download Ubuntu 12.04 "Precise Pangolin" x86_64 Minimal CD ISO Image mini.iso.

Start VMware Fusion and select Virtual Machine Library in Windows option on VMware Fusion toolbar. This will bring up Virtual Machine Library window showing all the Virtual Machine already available.


Virtual Machine Library
Click Add button and select New. This will bring up New Virtual Assistance Window showing Create New Virtual Machine.

Click Continue without disc as we will be using the downloaded ISO image. This will bring up Installation Media section.
Select Use operating system installation disc or image and click on arrows next to Choose a disc or disc image.... Select the Ubuntu ISO image and then click Continue.

Choose Operating System section should show Linux as Operating System and Ubuntu 64-bit as Version. Click Continue.

The Finish section will show Virtual Machine Summary, Click Finish. Select the location where we want to save the VM file and name the file.

A console window will be launched and OS install will start. Answered the prompts during the install process. Once update and reboot completes, a login prompt will appear. Log in to VM.


VM Login Screen

OpenSSH Server

After login, install OpenSSH Server to enable access to Ubuntu VM over SSH.

$sudo apt-get install openssh-server

Check whether SSH process is running.

$service ssh status

Either note down the IP address of VM from login screen (shown above) or using ifconfig command to be able to SSH into the VM remotely.

$ssh anil@172.16.191.158

You may need to remove SSH key if there is a fingerprint mismatch between the VM and remote client.

$ssh-keygen -R 172.16.191.158

Update Packages

You need to make sure all installed packages are current. Download any package updates and install.

anil@django:~$ sudo apt-get update
[sudo] password for anil: 
Hit http://us.archive.ubuntu.com precise Release.gpg
Get:1 http://us.archive.ubuntu.com precise-updates Release.gpg [198 B]
...

At this point VM is ready for installation of Python Virtualenv, Django, PostgreSQL, NGINX, and Gunicorn.

Python Virtualenv

Virtualenv is Virtual Python Environment builder to create separate Python environments. This enables to keep installations, dependencies, versions and permissions separate for different applications across different virtual environments.

Install python-virtualenv.

anil@django:~$ sudo apt-get install python-virtualenv
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  python-pip python-setuptools
The following NEW packages will be installed:
  python-pip python-setuptools python-virtualenv
...

Now, we need to create a virtual environment for our project (in this case, lendcafe) where we can install Python and Django packages.

anil@django:~$ sudo virtualenv /opt/lendcafe
New python executable in /opt/lendcafe/bin/python
Installing distribute...........................................................done.
Installing pip...............done.

You can name anything you like for your virtualenv.

Django

Django is a Python web framework. It enables rapid development of common web application tasks and adheres to DRY principle (Don't Repeat Yourself).

To install Django, first we need to activate virtualenv.

anil@django:~$ source /opt/lendcafe/bin/activate
(lendcafe)anil@django:~$ 

The activate script modifies shell prompt to show the currently active environment.

Install Django

(lendcafe)anil@django:~$ sudo pip install django
[sudo] password for anil: 
Downloading/unpacking django
  Downloading Django-1.6.2.tar.gz (6.6Mb): 6.6Mb downloaded
  Running setup.py egg_info for package django

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: django
  Running setup.py install for django
    changing mode of build/scripts-2.7/django-admin.py from 644 to 755

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /usr/local/bin/django-admin.py to 755
Successfully installed django
Cleaning up...

PostgreSQL

PostgreSQL is an open source object-relational database.

Deactivate the virtual environment.

(lendcafe)anil@django:~$ deactivate
anil@django:~$ 

Install Python dependencies for PostgreSQL

anil@django:~$ sudo apt-get install libpq-dev python-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
...

Install PostgreSQL

anil@django:~$ sudo apt-get install postgresql postgresql-contrib
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
...

NGINX

NGINX is an open source HTTP server and reverse proxy. It is known for high performance and low resource utilization. Instead of relying on threads to handle requests, it uses event-driven asynchronous architecture.

Install NGINX

anil@django:~$ sudo apt-get install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
...

Gunicorn

Gunicorn is a Python WSGI HTTP Server.

Activate virtualenv

anil@django:~$ source /opt/lendcafe/bin/activate
(lendcafe)anil@django:~$ 

Install Gunicorn within virtualenv

(lendcafe)anil@django:~$pip install gunicorn
Downloading/unpacking gunicorn
  Downloading gunicorn-18.0.tar.gz (366Kb): 366Kb downloaded
  Running setup.py egg_info for package gunicorn
  ...

Configuration

PostgreSQL

The default superuser for PostgreSQL is called postgres. We need to login as this user first.

anil@django:~$ sudo su - postgres
[sudo] password for anil: 
postgres@django:~$ 

The shell prompt should now show starting with postgres@....

Create a user. Answer the prompts. I decided to not make this user superuser, allow this new user to create databases or new roles.

postgres@django:~$ createuser -P
Enter name of role to add: anil
Enter password for new role: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

Create a new database. I named this database to be lendcafe.

postgres@django:~$ createdb lendcafe

To grant new user access to this database, first access the PostgreSQL interactive terminal and then grant all privileges. Type \q to quit.

postgres@django:~$ psql
psql (9.1.11)
Type "help" for help.
postgres=# GRANT ALL PRIVILEGES ON DATABASE lendcafe TO anil;
GRANT
postgres=# \q
postgres@django:~$ 

Test the new user log in to database. If you are not already logged in the system as new user, you will get error as shown below. Login as new user or use su command.

postgres@django:~$ psql -d lendcafe -U anil
psql: FATAL:  Peer authentication failed for user "anil"
postgres@django:~$ su - anil
Password: 
anil@django:~$ psql -d lendcafe -U anil
psql (9.1.11)
Type "help" for help.
lendcafe=> 
lendcafe=> \q
anil@django:~$ 

Django

To create a Django project, first switch to the virtualenv directory created during installation and activate the virtualenv.

anil@django:~$ cd /opt/lendcafe
anil@django:/opt/lendcafe$ source /opt/lendcafe/bin/activate
(lendcafe)anil@django:/opt/lendcafe$    

Start a new Django project. If you receive permission denied error as shown below, change the ownership of your environment directory. Then start a new Django project again. Check to make sure the project directory was created in virtualenv directory.

(lendcafe)anil@django:/opt/lendcafe$ django-admin.py startproject lcproject
CommandError: [Errno 13] Permission denied: '/opt/lendcafe/lcproject'
(lendcafe)anil@django:/opt/lendcafe$ sudo chown -R anil:anil /opt/lendcafe
[sudo] password for anil: 
(lendcafe)anil@django:/opt/lendcafe$ django-admin.py startproject lcproject
(lendcafe)anil@django:/opt/lendcafe$ ls -la
total 28
drwxr-xr-x 7 anil anil 4096 Feb 21 11:52 .
drwxr-xr-x 3 root root 4096 Feb 20 10:37 ..
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 bin
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 include
drwxrwxr-x 3 anil anil 4096 Feb 21 11:52 lcproject
drwxr-xr-x 3 anil anil 4096 Feb 20 10:37 lib
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 local
(lendcafe)anil@django:/opt/lendcafe$ 

For Django to be able to communicate with PosgreSQL database, we need to install PostgresSQL adapter for the Python Psycopg.

(lendcafe)anil@django:/opt/lendcafe$ pip install psycopg2
Downloading/unpacking psycopg2
  Downloading psycopg2-2.5.2.tar.gz (685Kb): 685Kb downloaded
  Running setup.py egg_info for package psycopg2

Installing collected packages: psycopg2
...
Successfully installed psycopg2
Cleaning up...
(lendcafe)anil@django:/opt/lendcafe$ 

Edit the setting.py file in directory and subdirectory named same as your project name.

(lendcafe)anil@django:/opt/lendcafe$ nano lcproject/lcproject/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'lendcafe',
        'USER': 'anil',
        'PASSWORD': 'myPassword',
        'HOST': 'localhost',
        'PORT': '',
    }
}

Run the following command to add Django specific database tables and configuration to database. I received the following error message when I tried to run the command.

(lendcafe)anil@django:/opt/lendcafe$ python lcproject/manage.py syncdb
Traceback (most recent call last):
  File "lcproject/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management

The StackOverflow discussion django import error - No module named core.management provides potential solutions to this error.

My issue turned out that I originally installed Django as root user while I was trying to run the above command as different user. When I ran the above command with sudo, I encountered a different error.

(lendcafe)anil@django:/opt/lendcafe/lcproject$ sudo python manage.py syncdb
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
...
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 25, in <module>
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named psycopg2

At this point, I decided to reinstall Django, this time without sudo.

(lendcafe)anil@django:/$ pip install django
Downloading/unpacking django
  Downloading Django-1.6.2.tar.gz (6.6Mb): 6.6Mb downloaded
  Running setup.py egg_info for package django

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: django
  Running setup.py install for django
    changing mode of build/scripts-2.7/django-admin.py from 664 to 775

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /opt/lendcafe/bin/django-admin.py to 775
Successfully installed django
Cleaning up...

Once I reinstalled Django, I was able to successfully add Django specific database tables.

(lendcafe)anil@django:/opt/lendcafe/lcproject$ python manage.py syncdb
Creating tables ...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'anil'): 
Email address: xyz@example.com
Password: 
Password (again): 
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
(lendcafe)anil@django:/opt/lendcafe/lcproject$ 

Gunicorn

Create a gunicorn_config.py file in /opt/lendcafe directory and add following entries in the file.

(lendcafe)anil@django:/opt/lendcafe$ nano gunicorn_config.py

command = '/opt/lendcafe/bin/gunicorn'
pythonpath = '/opt/lendcafe/lcproject'
bind = '127.0.0.1:8001'
workers = 5
user = 'anil'

Use the following command to run the server.

(lendcafe)anil@django:/opt/lendcafe$ /opt/lendcafe/bin/gunicorn -c /opt/lendcafe/gunicorn_config.py lcproject.wsgi
2014-02-21 15:41:13 [2892] [INFO] Starting gunicorn 18.0
2014-02-21 15:41:13 [2892] [INFO] Listening at: http://127.0.0.1:8001 (2892)
2014-02-21 15:41:13 [2892] [INFO] Using worker: sync
2014-02-21 15:41:13 [2897] [INFO] Booting worker with pid: 2897
2014-02-21 15:41:13 [2898] [INFO] Booting worker with pid: 2898
2014-02-21 15:41:13 [2899] [INFO] Booting worker with pid: 2899
2014-02-21 15:41:13 [2900] [INFO] Booting worker with pid: 2900
2014-02-21 15:41:13 [2901] [INFO] Booting worker with pid: 2901

Background the process by using ctrl + z and then typing bg followed by Enter. supervisord and screen can be used to manage Gunicorn start/restart. Refer to How to Install and Manage Supervisor on Ubuntu and Debian VPS for more information.

Enter the IP address and port specified in gunicorn_config.py file in a browser's address bar. If you see the message It worked! Congratulations on your first Django-powered page., you are good to go.


Welcome to Django GUI

NGINX

Django settings for the project are stored in settings.py file. As NGINX will be used for static files, modify the location of STATIC_URL in /opt/lendcafe/lcproject/lcproject/settings.py file.

(lendcafe)anil@django:/opt/lendcafe/lcproject/lcproject$ nano settings.py

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.6/howto/static-files/

STATIC_URL = '/opt/lendcafe/static/'

Create a subdirectory static in /opt/lendcafe/ directory.

(lendcafe)anil@django:/opt/lendcafe$ mkdir static
(lendcafe)anil@django:/opt/lendcafe$ ls -la
total 36
drwxr-xr-x 8 anil anil 4096 Feb 22 15:26 .
drwxr-xr-x 3 root root 4096 Feb 20 10:37 ..
drwxr-xr-x 2 anil anil 4096 Feb 21 15:28 bin
-rw-rw-r-- 1 anil anil  133 Feb 21 15:41 gunicorn_config.py
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 include
drwxrwxr-x 3 anil anil 4096 Feb 21 11:52 lcproject
drwxr-xr-x 3 anil anil 4096 Feb 20 10:37 lib
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 local
drwxrwxr-x 2 anil anil 4096 Feb 22 15:26 static
(lendcafe)anil@django:/opt/lendcafe$ 

NGINX is a reverse proxy and HTTP server. There are two blocks, Server-block and Location-blocks in configuration file that we will be primarily working with. The server-block is very similar to virtual host and location-block to URI. There is much more information available in NGINX Beginner's Guide.

Create a new NGINX config file lendcafe in /etc/nginx/sites-available/ directory and enter the following configuration information.

(lendcafe)anil@django:/opt/lendcafe$ sudo nano /etc/nginx/sites-available/lendcafe

server {
    server_name localhost;

    location /static/ {
        alias /opt/lendcafe/static/;
    }

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
    }
}

Create a symbolic link in /etc/nginx/sites-enabled directory to the NGINX configuration file lendcafe.

(lendcafe)anil@django:/etc/nginx/sites-enabled$ sudo ln -s /etc/nginx/sites-available/lendcafe

Delete the symbolic link default.

(lendcafe)anil@django:/etc/nginx/sites-enabled$ sudo rm default

Restart NGINX service to changes to take effect.

(lendcafe)anil@django:/etc/nginx/sites-enabled$ sudo service nginx restart
Restarting nginx: nginx.
(lendcafe)anil@django:/etc/nginx/sites-enabled$ 

VM is setup for Python/Django web development.

Tuesday, February 11, 2014

OpenStack: Virtual Image Instances using Horizon Dashboard

Install Addendum


Enable VT in BIOS

An addendum to install steps defined in my previous post OpenStack: Quick Install using DevStack is required to avoid a surprise that I encountered after the install. Please check to make sure BIOS is at latest version available from the system manufacturer and Intel's Virtualization Technology (VT) is enabled in BIOS.

anil@OSCloud:~$ sudo apt-get install cpu-checker
anil@OSCloud:~$ sudo kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

If CPU doesn't support VT, the output will show CPU does not support KVM extensions.
The OpenStack Horizon Dashboard is implemented as a Python/Django web application that provides admin and user interface to OpenStack services.

Horizon Dashboard


Log in

In web browser, type the IP address for the dashboard. On Log In page enter User Name and Password and click Sign In. When signing in as Admin, the home page shows the Admin panel - System Panel - Overview.

Horizon Dashboard Admin Home Page


Existing Virtual Machine Images

By clicking Images category in Admin - System panel on the left, a list of available images are viewed. In default installation, CirrOS x86_64 image is made available in AMI/ARI/AKI format.

CirrOS images are tiny cloud guest images with minimal Linux distribution that can also be downloaded from LaunchPad. The AMI/ARI/AKI is the image format supported by Amazon EC2. AMI (Amazon Machine Image) is a virtual machine raw image. ARI (Amazon Kernel Image) is a kernel file (vmlinuz) that will load initially to boot image. ARI (Amazon Ramdisk Image) is ramdisk file (initrd) mounted at boot time.

Horizon Dashboard Admin Images Page


Launch Instances

Clicking on the Project tab in left panel shows the overview of current project.

Horizon Dashboard Project Home Page

To launch an instance from an image, click Images and Snapshot category in Project - Manage Compute panel on the left.

Horizon Dashboard Project Images & Snapshot Page

Select an image and click Launch. A Launch Instance modal pop-up appear. Enter a name in Instance Name field in Details tab.

Horizon Launch Instance Details Popup

In Access & Security tab, enter a passphrase in Admin Pass and Confirm Admin Pass fields.

Horizon Launch Instance Access & Security Popup

Upon clicking Launch, Horizon dashboard switches to Project - Manage Compute - Instances page and shows the Instances running.

Horizon Project Instances

Clicking on Instance Name hyperlink shows the Instance Details for that specific instance with three tabs for Overview, Log and Console.

Horizon Project Instance Console

Though the Project - Manage Compute - Instances page shows instance to be Active and Running, the console for the instance is displaying an error message.

This kernel requires an x86-64 CPU, but only detected an i686 CPU.
Unable to boot - please use a kernel appropriate for your CPU.


Error Troubleshooting

A little bit of googling suggested to check whether the 64-bit PC (amd64, x86_64) or 32-bit PC (x86) version of host operating system is installed. Sure enough, the Ubuntu version installed on OSCloud host is x86 and not x86-64 version. I can't use x86-64 instance images on OSCloud host.

anil@OSCloud:~$ uname -a
Linux OSCloud 3.2.0-58-generic-pae #88-Ubuntu SMP Tue Dec 3 18:00:02 UTC 2013 i686 i686 i386 GNU/Linux

After terminating the newly created instance test1 and deleting all x86_64 Images, the next step was to either find or build x86 images and start a new x86 instance.

Prebuilt Virtual Machine Images

As OSCloud host is using QEMU Hypervisor, it made sense to look for qcow2 (QEMU copy-on-write) format x86 images. At CirrOS download page, I found a bootable qcow disk image for i386 and decided to try it out.

Create Images

To create images, on Admin tab, select Images and then click Create Image button in right pane. On Create An Image page, enter Name for the image, select Image Source, Image Location, and Format. Select the Public checkbox to make available this image to everyone. Then click Create Image. The image will be queued for creation.

Horizon Admin Create An Image

Once images are created, they will be available to launch instances in projects following the steps listed above in Launch Instances section.

Horizon Admin Images

Horizon Project Launch Instance

In next blog post, I will start to dig deeper into high level solution design using OpenStack. Your feedback and comments are welcome.

Tuesday, February 04, 2014

OpenStack: Quick Install using DevStack

Though not a recommended method for installing OpenStack for production, DevStack offers an easy method to install and run an OpenStack cloud either on hardware or even within virtual machine.

In this post, I walk through installing OpenStack using DevStack on an old Sony Vaio laptop on local network. The DevStack site provides instructions for installing OpenStack on Virtual Machines and on Hardware. The detailed instructions for installing OpenStack on a single hardware machine, that I followed, are available at DevStack.

Install Ubuntu Server OS

I repurposed a SONY VAIO laptop for DevStack install that was originally wiped clean with Darik's Boot and Nuke in preparation for disposal.

As DevStack downloads and installs all dependencies, I downloaded 64-bit PC (amd64, x86_64)Ubuntu 12.04 "Precise Pangolin" using Minimal CD mini.iso and burnt a CD on my MacBook Pro.

Booted Sony laptop using the Ubuntu minimal CD and selected Install from Installer boot menu, followed the prompts, and accepted default options for most prompts. It takes over an hour for the installation to download and install the base system.

Installed OpenSSH Server to enable access to Ubuntu server over SSH from my MBP.
$sudo apt-get install openssh-server
Checked whether SSH process is running by using one of the two commands listed below. The output shows process running.
anil@OSCloud:~$ ps aux | grep ssh
root      1394  0.0  0.0   6684  2416 ?        Ss   21:43   0:00 /usr/sbin/sshd -D   
anil      1534  0.0  0.0   4384   836 pts/0    S+   22:06   0:00 grep --color=auto ssh

anil@OSCloud:~$ service ssh status
ssh start/running, process 1394
Checked the IP address of Ubuntu server so that I can remotely access the server.
anil@OSCloud:~$ ifconfig
eth0    Link encap:Ethernet  HWaddr 00:1d:ba:23:9a:c5  
        inet addr:10.0.1.25  Bcast:10.0.1.255  Mask:255.255.255.0
        inet6 addr: fe80::21d:baff:fe23:9ac5/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:875 errors:0 dropped:0 overruns:0 frame:0
        TX packets:202 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000 
        RX bytes:522168 (522.1 KB)  TX bytes:19796 (19.7 KB)
        Interrupt:16 

lo      Link encap:Local Loopback  
        inet addr:127.0.0.1  Mask:255.0.0.0
        inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING  MTU:16436  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0 
        RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
While trying to access Ubuntu server from MPB over SSH, I received following error.
ANILs-MacBook-Pro:~ anilgupta$ ssh anil@10.0.1.25
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
26:54:a3:4e:cd:a3:6c:80:f3:36:2c:b3:c9:17:f0:db.
Please contact your system administrator.
Add correct host key in /Users/anilgupta/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/anilgupta/.ssh/known_hosts:6
RSA host key for 10.0.1.25 has changed and you have requested strict checking.
Host key verification failed.
This is well-known error when there is fingerprint mismatch between the host (Ubuntu Server) and client (MBP). In this case, as there is no malicious attempt, I removed the offending key from MBP using ssh-keygen -R command.
ANILs-MacBook-Pro:~ anilgupta$ ssh-keygen -R 10.0.1.25
# Host 10.0.1.25 found: line 6 type RSA
/Users/anilgupta/.ssh/known_hosts updated.
Original contents retained as /Users/anilgupta/.ssh/known_hosts.old
After removing the offending key, I was able to successfully SSH into Ubuntu Server.
ANILs-MacBook-Pro:~ anilgupta$ ssh anil@10.0.1.25
The authenticity of host '10.0.1.25 (10.0.1.25)' can't be established.
RSA key fingerprint is 26:54:a3:4e:cd:a3:6c:80:f3:36:2c:b3:c9:17:f0:db.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.25' (RSA) to the list of known hosts.
anil@10.0.1.25's password: 
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-58-generic-pae i686)

* Documentation:  https://help.ubuntu.com/
Last login: Tue Jan 28 21:40:01 2014
Gave sudo privileges to anil user account
anil@OSCloud:~$ sudo visudo
A file /etc/sudoers.tmp was opened in nano editor. Added the following line to the opened file and then saved using ^X.
anil    ALL=(ALL) NOPASSWD: ALL
At this point, the Ubuntu server is ready for DevStack install.

Install DevStack

Before installing DevStack, it is good idea to review DevStack GitHub ReadMe file. A few points that stood out in this file for me were:
  • Be sure to carefully read stack.sh and any other scripts your execute before you run them, as they install software and will alter your networking configuration.
  • The DevStack master branch generally points to trunk versions of OpenStack components. For older, stable versions, look for branches names stable/[release] in DevStack repo.
  • You can also pick specific OpenStack project releases by setting the appropriate *_BRANCH variables in the localrc section of local.conf.
  • You can override environment variables used in stack.sh by creating file name local.conf with a localrc section.
  • Swift is disabled by default. When, it is configured with only one replica to avoid being IO/memory intensive.
The last point was particularly important for my install as I really would like to understand storage components in OpenStack.
Install Git distributed version control system.
anil@OSCloud:~$ sudo apt-get install git -y
[sudo] password for anil: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
    git-man liberror-perl patch
Suggested packages:
    git-daemon-run git-daemon-sysvinit git-doc git-el git-arch git-cvs git-svn
    git-email git-gui gitk gitweb diffutils-doc
The following NEW packages will be installed:
    git git-man liberror-perl patch
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 6,703 kB of archives.
After this operation, 15.5 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main liberror-perl all 0.17-1 [23.8 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main git-man all 1:1.7.9.5-1 [630 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main git i386 1:1.7.9.5-1 [5,963 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ precise/main patch i386 2.6.1-3 [86.0 kB]
Fetched 6,703 kB in 16s (409 kB/s)                                             
Selecting previously unselected package liberror-perl.
Reading database ... 46735 files and directories currently installed.)
Unpacking liberror-perl (from .../liberror-perl_0.17-1_all.deb) ...
Selecting previously unselected package git-man.
Unpacking git-man (from .../git-man_1%3a1.7.9.5-1_all.deb) ...
Selecting previously unselected package git.
Unpacking git (from .../git_1%3a1.7.9.5-1_i386.deb) ...
Selecting previously unselected package patch.
Unpacking patch (from .../patch_2.6.1-3_i386.deb) ...
Processing triggers for man-db ...
Setting up liberror-perl (0.17-1) ...
Setting up git-man (1:1.7.9.5-1) ...
Setting up git (1:1.7.9.5-1) ...
Setting up patch (2.6.1-3) ...
Clone the DevStack repository from GitHub.
anil@OSCloud:~$ git clone https://github.com/openstack-dev/devstack.git
Cloning into 'devstack'...
remote: Reusing existing pack: 14896, done.
remote: Total 14896 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (14896/14896), 3.55 MiB | 1.59 MiB/s, done.
Resolving deltas: 100% (10138/10138), done.
Change to devstack directory.
anil@OSCloud:~$ cd devstack
anil@OSCloud:~/devstack$ ls -la
total 260
drwxr-xr-x 11 anil anil  4096 Jan 29 12:15 .
drwxr-xr-x  4 anil anil  4096 Jan 29 12:15 ..
-rw-rw-r--  1 anil anil  1857 Jan 29 12:15 AUTHORS
-rwxrwxr-x  1 anil anil  2871 Jan 29 12:15 clean.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 driver_certs
-rw-rw-r--  1 anil anil  1552 Jan 29 12:15 eucarc
-rw-rw-r--  1 anil anil  1145 Jan 29 12:15 exerciserc
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 exercises
-rwxrwxr-x  1 anil anil  1962 Jan 29 12:15 exercise.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 extras.d
drwxrwxr-x  7 anil anil  4096 Jan 29 12:15 files
-rw-rw-r--  1 anil anil 66652 Jan 29 12:15 functions
drwxrwxr-x  8 anil anil  4096 Jan 29 12:15 .git
-rw-rw-r--  1 anil anil   226 Jan 29 12:15 .gitignore
-rw-rw-r--  1 anil anil    81 Jan 29 12:15 .gitreview
-rw-rw-r--  1 anil anil 11243 Jan 29 12:15 HACKING.rst
drwxrwxr-x  6 anil anil  4096 Jan 29 12:15 lib
-rw-rw-r--  1 anil anil 10143 Jan 29 12:15 LICENSE
-rw-rw-r--  1 anil anil   259 Jan 29 12:15 .mailmap
-rw-rw-r--  1 anil anil  3343 Jan 29 12:15 openrc
-rw-rw-r--  1 anil anil 14473 Jan 29 12:15 README.md
-rwxrwxr-x  1 anil anil   638 Jan 29 12:15 rejoin-stack.sh
-rwxrwxr-x  1 anil anil   870 Jan 29 12:15 run_tests.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 samples
-rw-rw-r--  1 anil anil 12425 Jan 29 12:15 stackrc
-rwxrwxr-x  1 anil anil 43835 Jan 29 12:15 stack.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 tests
drwxrwxr-x  6 anil anil  4096 Jan 29 12:15 tools
-rwxrwxr-x  1 anil anil  3769 Jan 29 12:15 unstack.sh
Make a copy of local.conf file from devstack/samples folder to devstack folder
anil@OSCloud:~/devstack$ cp samples/local.conf local.conf
Open local.conf file in editor and add the listed configuration changes in [[local|localrc]] section. I chose to have same password as ADMIN for other accounts. I also chose to enable Swift. Save the file.
anil@OSCloud:~/devstack$ nano local.conf

ADMIN_PASSWORD = select_password
MYSQL_PASSWORD = select_password
RABBIT_PASSWORD = select_password
SERVICE_PASSWORD = $ADMIN_PASSWORD

enable_service s-proxy s-object s-container s-account

Run DevStack using stack.sh.

anil@OSCloud:~/devstack$ ./stack.sh
[sudo] password for anil: 
Using mysql database backend

################################################################################
ENTER A SERVICE_TOKEN TO USE FOR THE SERVICE ADMIN TOKEN.
################################################################################
This value will be written to your localrc file so you don't have to enter it 
again.  Use only alphanumeric characters.
If you leave this blank, a random default value will be used.
Enter a password now:
select_password
2014-01-29 12:48:57 + echo_summary 'Installing package prerequisites'
...
Once install finishes, access information for the OpenStack cloud is displayed.
Horizon is now available at http://10.0.1.25/
Keystone is serving at http://10.0.1.25:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: select_password
This is your host ip: 10.0.1.25
stack.sh completed in 339 seconds.
The OpenStack logs are stored in /opt/stack/logs directory. The latest detailed log can be accessed using symlink /opt/stack/logs/stack.sh.log. Unless there are errors, the latest summary log symlink /opt/stack/logs/stack.sh.log.summary is sufficient to check for installed, configured, and status of services.
anil@OSCloud:~/devstack$ cat /opt/stack/logs/stack.sh.log.summary
stack.sh log /opt/stack/logs/stack.sh.log.2014-01-29-134716
Installing package prerequisites
Installing OpenStack project source
Installing Tempest
Starting RabbitMQ
Configuring and starting MySQL
Starting Keystone
Configuring and starting Horizon
Configuring Glance
Configuring Swift
Configuring Cinder
Configuring Nova
Starting Swift
Starting Glance
Starting Nova API
Starting Nova
Starting Cinder
Uploading images
Initializing Tempest
stack.sh completed in 346 seconds.
At this point, I was able to log in to Horizon Dashboard using http://10.0.1.25/ from MBP.

OpenStack Dashboard

Miscellaneous Operations


Stop DevStack

unstack.sh stops the processes that were started by stack.sh. All processes can be stopped by setting UNSTACK_ALL or specifying --all on the command line.
anil@OSCloud:~/devstack$ ./unstack.sh --all
    * Stopping web server apache2                                                                                                                       apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
    ... waiting                                                                          [ OK ]
tgt stop/waiting
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
tgtd seems to be in a bad state, restarting...
stop: Unknown instance: 
tgt start/running, process 13058
tgt stop/waiting
mysql stop/waiting
Stopping rabbitmq-server: rabbitmq-server.

Restart DevStack

Bring volume group online so that cinder-volume will start without errors
sudo losetup -f /opt/stack/data/stack-volumes-backing-file
Use rejoin-stack.sh to restart the DevStack
./rejoin-stack.sh &

Reboot Host

After rebooting the Host, when I tried to login to dashboard, I received the error An error occurred authenticating. Please try again later. The /var/log/apache2/horizon_error.log only shows Login failed for user "admin" as entry. Checked the SELINUX status.
anil@OSCloud:~/devstack$ sestatus
SELinux status:                 disabled
The reason turned out that I needed to restart Devstack by using the command listed above in Restart DevStack section.

In next post, I will take the Dashboard for a spin.


Digital Ocean

Tuesday, January 28, 2014

OpenStack: Overview of Service Components

OpenStack is made up of several projects that together build up the OpenStack cloud. Three core components are Compute, Storage, and Network.

OpenStack Training Guides provide a nice conceptual architecture of OpenStack service components, created by Solinea.

OpenStack Conceptual Architecture

Compute (Nova)

Nova services provide computing resources through virtual machine (VM) instances and works with several virtualization technologies, such as KVM, QEMU, VMWare ESX, Xen, Hyper-V, and LXC. A complete list of supported Hypervisors is listed at OpenStack Wiki.

Nova is also called Cloud Controller as it provides framework for provisioning and managing VMs. Nova also provides ephemeral storage.

With Nova services, IT departments can offer private cloud services to internal departments that can scale computing resources on-demand as workload varies. It is analogous to Amazon EC2.
Nova is made up of several service components:
  • nova-compute: Runs the VM instances.
  • nova-scheduler: Decides which host will run the requested instance.
  • nova-api: Service interface to Nova, such as a call to start up a Nova instance.
  • nova-network: Network services
  • nova-objectstore: File storage services
  • nova-common: The underlying common libraries
  • nova-cert: Certificate management service used to authenticate to Nova

Object Storage (Swift)

By default, all storage disappears when VM instance is terminated. Swift services provide cost-effective scale-out redundant persistent storage to VM instances and responsible for ensuring data replication and integrity. The object storage is suitable for static data and stored as Objects. The objects are stored and replicated on disks spread across storage cluster nodes. The stored data persists until deleted by users.

Swift is analogous to Amazon S3. Swift provides RESTful API for integration with other application, storage and services.

Swift is made of several service components:
  • swift-proxy: Accepts, authorizes, authenticates incoming requests.
  • swift-account: Manages database of accounts.
  • swift-container: Contains mapping of containers.
  • swift-object: Contains mapping of objects.

Block Storage (Cinder)

Cinder provides high performance persistent block storage for use with VM instances. Cinder volumes are appropriate for database, file system and raw block storage. Cinder is supported by several storage platforms including Ceph, Nexenta, SolidFire, Zadara, CloudByte, Coraid and Scality to name a few startups in this area. A complete lists of storage devices supported by Cinder drivers is available at Cinder Support Matrix.

Cinder is analogous to Amazon EBS. It also provides snapshot management with capability to either restore snapshot or use snapshot to create a new block storage volume.
Following services are available with Cinder:
  • cinder-api: Authenticates and routes requests to block storage.
  • cinder-scheduler: Scheduling and routing requests to volume service.
  • cinder-volume: Managing back-end block storage devices.
  • cinder-backup: Backup Cinder volume to Swift.

Network (Neutron)

Neutron (formerly Quantum) provides virtual network service for connectivity and addressing used by other services. OpenStack networking can have multiple private networks with overlapping IP addressing schemes and relies on Keystone for authentication and authorization for API access. A standard implementation includes management network, data network, external network, and API network. Though started with basic Linux VLANs and IP tables, it now includes plugins which are pluggable back-end implementation of OpenStack Networking API: Open vSwitch, Linux Bridge, NEC OpenFlow, etc.

Following services are available with Networking:
  • quantum-server: Passing user requests to the configured Plugin for processing.
  • plugin agent (quantum-*-plugin-agent): Perform local vSwitch configuration on hypervisor.
  • dhcp agent (quantum-dhcp-agent): Provides DHCP services to internal networks.
  • l3 agent (quantum-l3-agent): Provides L3/NAT forwarding for access to external network.

Image (Glance)

Glance provides catalog and repository for disk, server and snapshot images. Glance API service is used for discovery, registration and delivery of images. Users can choose from available images, create their own from existing servers and start new compute instances from base templates. Glance supports multi-format image registry including raw, AMI (machine), QCOW2 (QEMU/KVM), VMDK (VMWare), VHD (Hyper-V), VDI (VirtualBox), and OVF.

Glance is made of several service components:
  • glance-api: Accept incoming API requests and communicate with other components.
  • glance-registry: Stores and retrieves metadata about images.
  • image store: Stores actual image blobs, can be local filesystem or Swift.

Identity (Keystone)

Keystone provides authentication authorization for managing users, accounts and role information for all OpenStack services. It also provides catalog of available services. Keystone provides an authentication token that is passed between services after validation for users to be able to use OpenStack resources. Keystone can also integrate with existing backend director services like LDAP.

Dashboard (Horizon)

Horizon provides a web-based graphical interface to access, provision and automate resources for administrators and users. It is implemented as Python/WSGI/Django web application running on Apache Web Server. As an alternate, custom command line scripts can be developed using OpenStack API.

In the next blog post, I plan to perform a quick install of OpenStack.

Thursday, January 23, 2014

OpenStack: The Building Block for Private Cloud

I am back after an extended break from blogging about data storage topics. As I am no longer working at Quantum, I am free to blog about the recent developments in data storage without any concerns.

Last year, I started using cloud servers on Digital Ocean for PeerCube and worked with Amazon Web Services during Coursera's Introduction to Data Science course. Since then, I became very interested in private clouds and methods to establish and manage them.

With the rise of server virtualization, users and applications ability to spin up and spin down pre-built images as needed, and the success of Amazon Web Services (AWS) public cloud, there are fewer reasons for IT administrators to actively manage pooling and allocation of IT infrastructure resources. While there are several commercial (VMware vCloud) and open source platforms (Apache CloudStack, Eucalyptus, OpenStack) that let you build Infrastructure as a Service (IaaS) private cloud, I am particularly excited with the developments happening in OpenStack community.

I was sold on OpenStack as soon as I was able to install a DevStack environment on an old laptop and spin up images within couple of hours. Since then, I have been exploring OpenStack documentation and source code to understand this platform better with private cloud as a specific application in mind.

OpenStack Components

The modularized architecture of OpenStack includes following service components:
  1. Compute (Nova)
  2. Object Store (Swift)
  3. Block Storage (Cinder)
  4. Neutron, formerly Quantum (Network)
  5. Image (Glance)
  6. Identity (Keystone)
  7. Dashboard (Horizon)
In the next blog post, I will elaborate on each services components. Being from data storage industry, I am very interested in learning about the storage service components of OpenStack in-depth.

As I realized that I learn best by writing and sharing, future blog posts will be my journey toward understanding OpenStack and how to use it to establish and manage private clouds.

Book Resources