vi and terminal

For most parts vim is an amazing tool. It has a huge initial learning curve but the functionality it provides are simply amazing.
The only other software which I have heard a lot of good stories about is Emacs.
Vim has this amazing functionality where you can set vi on terminal

 set -o vi  

This feature helps you to use vi when trying to type out long commands on the terminal. Or better you could edit the long command on vi and come back to terminal to execute the command all in a bunch of simple commands which I plan to explain here.

  Warning

vim on terminal is similar to vim editor you can only type commands on your terminal using the INSERT MODE. Now that's awesome because you also get the command functionalities which you get in vim. "/" this means search on your existing command. So if you have a huge command which you want to search through use the "/" command modes.

  Use case

For the meat of vim usage. Say you are running a curl command and inserting a huge json file and you are doing it in command line.
If you have been in this situation before you feel my pain. With Vim mode things become a bit easier. First get inside the vi editor.

• Open a terminal
• Type vi in shell without going into insert mode (just type vi in shell)
• This takes you inside a temp vi file. Now you can use normal vi commands.
• Copy paste your long curl command in here. Make the necessary changes.
• Now save the file ":w" and quite ":q"
• Wala Vi runs the command in your terminal.

This is a powerful feature.

  Caveat and Fix

For some reason vi environment resets auto complete with tabspace which is quite annoying. After some searching I came across this q&a. This solution worked for me so I ended up using this as an alias in my ~/.profile. Makes my life super easy now. Just use the alias when I want to get inside vim mode.

 alias ii="set -o vi; bind '"\C-i": menu-complete'"  

How to use pycurl to provide status bar and percentage using python

Today I was tasked to write a python script that uses pycurl to upload a 30GB binary file to a cloud storage portal. The problem is using curl doesn't provide useful information such as progress and percentage completed. This is very useful if you want to redo parts of the artifact. Since our cloud storage supports partitioned uploads it becomes all the more important to upload in parts and provide the percentage uploaded. I used pycurl documentation to figure out most of the args that I needed to set for the upload. However the part which I felt most cute about was the progress bar. The way I hooked it up to the pycurl was using call back mechanism which pycurl provides Disclaimer: This code has been tested out in Redhat Linux v6 machine.

import pycurl
import os, sys

# pretty print progress and percentage completed
def progress(total_to_download, total_downloaded, total_to_upload, total_uploaded):
  if total_to_upload:
    percent_completed = float(total_uploaded)/total_to_upload       # You are calculating amount uploaded
    rate = round(percent_completed * 100, ndigits=2)                # Convert the completed fraction to percentage
    completed = "#" * int(rate)                                     # Calculate completed percentage
    spaces = " " * ( 100 - completed)                               # Calculate remaining completed rate      
    sys.stdout.write('[%s%s] %s%%' %(completed, spaces, rate))      # the pretty progress [####     ] 34%  
    sys.stdout.flush()


def upload_to_cloud(url, filename, is_proxy=False):
  if not os.path.exists(filename):
    raise Exception('did not find file')

  # initialize py curl
  c = pycurl.Curl()
  c.setopt(pycurl.UPLOAD, 1)
  if is_proxy:
    c.setopt(pycurl.PROXY, 'XXX')
    c.setopt(pycurl.PROXYPORT, 80)
  
  #For authenticated cloud store
  c.setopt(pycurl.USERPWD, 'XXXX' + ':' + 'XXXX')
  c.setopt(pycurl.READFUNCTION, open(filename, 'rb').read)
  c.setopt(pycurl.VERBOSE, 0)
  c.setopt(pycurl.URL, url)
  c.setopt(pycurl.NOPROGRESS, 0)
  c.setopt(pycurl.PROGRESSFUNCTION, progress)

  #Set size of the file to be uploaded
  filesize = os.path.getsize(filename)      # you can simply open the file and do a byte counter for this. Initially that's what I did then moved to os API
  c.setopt(pycurl.INFILESIZE, filezie)
  
  # Start transfer
  print 'Uploading file %s to url %s' %(filename, url)
  c.perform()         # this kicks off the pycurl module with all options set.
  c.close()


if __name__=='__main__':
  if len(sys.argv) < 2:
    print 'Usage python upload_to_cloud URL FILE_PATH'

  upload_to_cloud(sys.argv[1], sys.argv[2])