<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>lab.joelGillman.com &#187; shell</title>
	<atom:link href="http://lab.joelgillman.com/archives/tag/shell/feed" rel="self" type="application/rss+xml" />
	<link>http://lab.joelgillman.com</link>
	<description>Sketches, Code Bites and Toys</description>
	<lastBuildDate>Tue, 12 Jul 2011 15:03:29 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>Download a huge file using split and cat</title>
		<link>http://lab.joelgillman.com/archives/179_download-a-huge-file-using-split-and-ca</link>
		<comments>http://lab.joelgillman.com/archives/179_download-a-huge-file-using-split-and-ca#comments</comments>
		<pubDate>Thu, 14 Apr 2011 14:39:20 +0000</pubDate>
		<dc:creator>joel gillman</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[cli]]></category>
		<category><![CDATA[command line]]></category>
		<category><![CDATA[shell]]></category>
		<category><![CDATA[ssh]]></category>
		<category><![CDATA[terminal]]></category>

		<guid isPermaLink="false">http://lab.joelgillman.com/?p=179</guid>
		<description><![CDATA[I&#8217;ve got a decent internet connection at home sitting around 15 Mbps down. Not top of the line, but respectable nonetheless. I have a personal server that I use to store various files that generally range in size from 150 MB to 14 GB. Now if we lived in a perfect world the download would [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve got a decent internet connection at home sitting around 15 Mbps down. Not top of the line, but respectable nonetheless.</p>
<p>I have a personal server that I use to store various files that generally range in size from 150 MB to 14 GB. Now if we lived in a perfect world the download would take up as much bandwidth as it can which would have me downloading at around 1.8 MB/s. Unfortunately through forces that I understand to be outside my control I only get about 200 Kb/s.</p>
<p>Interestingly, I can download several files from the same sever concurrently at that speed. This is great when downloading several smaller files at the same time, but rather useless when you&#8217;ve got one big one. <em>It doesn&#8217;t seem fair.</em><span id="more-179"></span></p>
<p>This is where split and cat come in! These are two commands you can use via command line (though there are many GUI alternatives, but that&#8217;s lame). Split, of course, will take in a file and <em>split</em> it into smaller pieces. Cat (short for <em>concatenate</em>) will take all those pieces and reassemble them on the other side. Don&#8217;t be afraid, it&#8217;s super easy.</p>
<p>First, on the server, split your large file into a bunch of lil&#8217; pieces.</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #c20cb9; font-weight: bold;">split</span> <span style="color: #660033;">-b</span> 500m totally.legit.file.mkv my_split_file_</div></div>
<p>This will split &#8216;totally.legit.file.mkv&#8217; into 500 Mb pieces. The last piece will be less than 500 Mb. Your split up files will be named &#8216;my_split_file_aa&#8217;, &#8216;my_split_file_ab&#8217;, &#8216;my_split_file_ac&#8217; and so on.</p>
<p>Download these files to your local computer.</p>
<p>Now we&#8217;ll concatenate the files back together!</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #c20cb9; font-weight: bold;">cat</span> my_split_file_<span style="color: #000000; font-weight: bold;">*</span> <span style="color: #000000; font-weight: bold;">&gt;</span> totally.legit.file.mkv</div></div>
<p>Depending on how big the file is this could take a little while. When it&#8217;s done the split files will still be there and you&#8217;ll have a shiny new &#8216;totally.legit.file.mkv&#8217; ready to go!</p>
<p>Don&#8217;t forget to delete the split file pieces that are still laying around on your server and local computers.</p>
]]></content:encoded>
			<wfw:commentRss>http://lab.joelgillman.com/archives/179_download-a-huge-file-using-split-and-ca/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>iSightCapture, now with more SSH!</title>
		<link>http://lab.joelgillman.com/archives/130_isightcapture-now-with-more-ssh</link>
		<comments>http://lab.joelgillman.com/archives/130_isightcapture-now-with-more-ssh#comments</comments>
		<pubDate>Thu, 30 Apr 2009 18:28:50 +0000</pubDate>
		<dc:creator>joel gillman</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[automation]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[isight]]></category>
		<category><![CDATA[scp]]></category>
		<category><![CDATA[screencapture]]></category>
		<category><![CDATA[script]]></category>
		<category><![CDATA[shell]]></category>
		<category><![CDATA[ssh]]></category>
		<category><![CDATA[ssh-keys]]></category>
		<category><![CDATA[terminal]]></category>

		<guid isPermaLink="false">http://lab.joelgillman.com/?p=130</guid>
		<description><![CDATA[I think this kind of thing is pretty awesome. Take a picture from your iSight. Take a screencapture. Upload them both to a remote server. WHY DOESN&#8217;T EVERYONE WANT TO DO THIS?! Maybe you do, now you can (hopefully). Firstly, props to Dylan O&#8217;Donnell, who wrote the original script. I was personally unsatisfied with FTP [...]]]></description>
			<content:encoded><![CDATA[<p>I think this kind of thing is pretty awesome.  Take a picture from your iSight.  Take a screencapture.  Upload them both to a remote server. WHY DOESN&#8217;T EVERYONE WANT TO DO THIS?!  Maybe you do, now you can (hopefully).<span id="more-130"></span></p>
<p>Firstly, props to Dylan O&#8217;Donnell, who wrote the original script.  I was personally unsatisfied with FTP because it&#8217;s insecure and I always run into technical troubles with it. Probably because I never took the time to learn what I was doing with FTP via command line&#8230; But I favor SCP for my file transferring needs.<br/><br/></p>
<p>I also added a screen cap functionality because there&#8217;s all kinds of cool things you can do with that down the line.  Time-lapse animation, security, you name it!<br/><br/></p>
<p>If you want the iSight captures to work you must download and install <a href='http://lab.joelgillman.com/thepress/wp-content/uploads/2009/04/isightcapture.zip'>isightcapture</a>! I&#8217;ve installed it to /usr/sbin/ and hardcoded the script to look there. If you don&#8217;t know what you&#8217;re doing, I suggest you do the same.<br/><br/></p>
<p><strong>&#8220;Installing&#8221; isightcapture (assuming it&#8217;s unzipped and on your desktop):</strong></p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">user$ </span><span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #c20cb9; font-weight: bold;">cp</span> ~<span style="color: #000000; font-weight: bold;">/</span>Desktop<span style="color: #000000; font-weight: bold;">/</span>isightcapture <span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>sbin<span style="color: #000000; font-weight: bold;">/</span></div></div>
<p>Quick rundown of what&#8217;s happening here:<br />
- Grab the date and time info and prepare the filenames (YearMonthDayHourMinuteSecond)<br />
- Take a picture with the iSight, save it to a folder (~/Documents/pics/)<br />
- Take a screencap, save it to the same folder (~/Documents/pics/)<br />
- Resize all of the jpgs in the pics folder so they&#8217;re no more than 800px in any dimention<br />
- Check to see if the computer has a network connection, if it does upload the files to a remote server (via scp) and delete the local images. If there&#8217;s no connection, do nothing (images will try to upload next time the script is run)</p>
<p>In order for this to work <strong><em>you must have SSH Keys setup with the server you are connecting to</em></strong>. If you don&#8217;t it will not work.  If you google &#8220;ssh keys&#8221; there are a million ways to do it but this is the site that actually got it working on my modern mac: <a href="http://www.webficient.com/2007/11/16/mac-and-ssh-keys">http://www.webficient.com/2007/11/16/mac-and-ssh-keys</a><br/><br/></p>
<p>Here&#8217;s a quick and dirty run down of how to do that, but I&#8217;m just copying from <a href="http://www.webficient.com/2007/11/16/mac-and-ssh-keys">Phil&#8217;s post</a> so don&#8217;t blame me if it doesn&#8217;t work for you :)<br/><br/></p>
<p>Generate the ssh key (obviously replace the CAPS stuff with your info). When asked for a passphrase just hit enter, this will make an empty passphrase which, I admit, is very insecure but it&#8217;s better than nothing and the script won&#8217;t work with a passphrase :( :</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">user$ </span><span style="color: #c20cb9; font-weight: bold;">ssh-keygen</span> <span style="color: #660033;">-t</span> rsa <span style="color: #660033;">-f</span> ~<span style="color: #000000; font-weight: bold;">/</span>.ssh<span style="color: #000000; font-weight: bold;">/</span>id_rsa</div></div>
<p>If you&#8217;ve never used SSH to make an outgoing connect on the machine you are connecting to you will also need to create the ssh folder on that machine and make it so only your user can read it:</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">user$ </span><span style="color: #c20cb9; font-weight: bold;">ssh</span> USER<span style="color: #000000; font-weight: bold;">@</span>YOUR.SERVER.COM <span style="color: #ff0000;">'mkdir ~/.ssh;chmod 700 ~/.ssh'</span></div></div>
<p>Now copy your public key (id_rsa.pub) to the server you&#8217;re connecting to (I always name the copy on the server &#8220;temp_pub_key&#8221; so I know exactly what file I need):</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">user$ </span><span style="color: #c20cb9; font-weight: bold;">scp</span> ~<span style="color: #000000; font-weight: bold;">/</span>.ssh<span style="color: #000000; font-weight: bold;">/</span>id_rsa.pub USER<span style="color: #000000; font-weight: bold;">@</span>YOUR.SERVER.COM:~<span style="color: #000000; font-weight: bold;">/</span>.ssh<span style="color: #000000; font-weight: bold;">/</span>temp_pub_key</div></div>
<p>Now connect to the remote server and navigate to ~/.ssh/</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">remote-user$ </span><span style="color: #7a0874; font-weight: bold;">cd</span> ~<span style="color: #000000; font-weight: bold;">/</span>.ssh<span style="color: #000000; font-weight: bold;">/</span></div></div>
<p>On the server, check to see if the file &#8220;authorized_keys&#8221; exists, if it doesn&#8217;t rename your &#8220;temp_pub_key&#8221; to &#8220;authorized_keys:</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">remote-user$ </span><span style="color: #c20cb9; font-weight: bold;">mv</span> temp_pub_key authorized_keys</div></div>
<p>If &#8220;authorized_keys&#8221; already exists then you want to add your public key to the end of the file:</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666;">remote-user$ </span><span style="color: #c20cb9; font-weight: bold;">cat</span> temp_pub_key <span style="color: #000000; font-weight: bold;">&amp;</span>gt;<span style="color: #000000; font-weight: bold;">&amp;</span>gt; authorized_keys</div></div>
<p>If &#8220;authorized_keys&#8221; didn&#8217;t already exists, you&#8217;ll also have to set the permissions of the authorized_keys file:</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #c20cb9; font-weight: bold;">ssh</span> USER<span style="color: #000000; font-weight: bold;">@</span>YOUR.SERVER.COM <span style="color: #ff0000;">'chmod 644 ~/.ssh/authorized_keys2'</span></div></div>
<p>Here is the automation script originally written by Dylan O&#8217;Donnell with my tweaks and editing for SSH and screencaps.</p>
<div class="codecolorer-container bash default" style="overflow:auto;white-space:nowrap;border:1px solid #9F9F9F;width:435px;height:300px;"><div class="bash codecolorer" style="padding:5px;font:normal 12px/1.4em Monaco, Lucida Console, monospace;white-space:nowrap"><span style="color: #666666; font-style: italic;">#!/bin/bash</span><br />
<span style="color: #666666; font-style: italic;">#</span><br />
<span style="color: #666666; font-style: italic;"># Original iSight Auto Upload Script - by Dylan O'Donnell 2006</span><br />
<span style="color: #666666; font-style: italic;"># Edited for ssh and made more awesome by Joel Gillman 2009 </span><br />
<span style="color: #666666; font-style: italic;">#####################################################</span><br />
<span style="color: #666666; font-style: italic;"># Edit CAPITALS with appropriate details </span><br />
<span style="color: #666666; font-style: italic;"># &nbsp;Notes : You must have ssh-keys setup with the server you are </span><br />
<span style="color: #666666; font-style: italic;"># &nbsp; &nbsp; &nbsp; &nbsp;: uploading to before hand. Otherwise this will not work.</span><br />
<span style="color: #666666; font-style: italic;"># &nbsp; &nbsp; &nbsp; &nbsp;:</span><br />
<span style="color: #666666; font-style: italic;"># &nbsp; &nbsp; &nbsp; &nbsp;: Try here for tutorial on ssh-keys (worked for me)</span><br />
<span style="color: #666666; font-style: italic;"># &nbsp; &nbsp; &nbsp; &nbsp;: &nbsp;http://www.webficient.com/2007/11/16/mac-and-ssh-keys</span><br />
<span style="color: #666666; font-style: italic;">#</span><br />
<span style="color: #666666; font-style: italic;">#####################################################</span><br />
<br />
<span style="color: #666666; font-style: italic;"># Generate filename based on date stamp</span><br />
<span style="color: #007800;">picture</span>=$<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span> +<span style="color: #000000; font-weight: bold;">%</span>y<span style="color: #000000; font-weight: bold;">%</span>m<span style="color: #000000; font-weight: bold;">%</span>d<span style="color: #000000; font-weight: bold;">%</span>H<span style="color: #000000; font-weight: bold;">%</span>M<span style="color: #000000; font-weight: bold;">%</span>S<span style="color: #7a0874; font-weight: bold;">&#41;</span>_pic.jpg;<br />
<span style="color: #007800;">screen</span>=$<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span> +<span style="color: #000000; font-weight: bold;">%</span>y<span style="color: #000000; font-weight: bold;">%</span>m<span style="color: #000000; font-weight: bold;">%</span>d<span style="color: #000000; font-weight: bold;">%</span>H<span style="color: #000000; font-weight: bold;">%</span>M<span style="color: #000000; font-weight: bold;">%</span>S<span style="color: #7a0874; font-weight: bold;">&#41;</span>_scr.jpg;<br />
<br />
<span style="color: #666666; font-style: italic;"># Take iSight Photo and store with datestamp filename</span><br />
<span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>sbin<span style="color: #000000; font-weight: bold;">/</span>isightcapture <span style="color: #660033;">-w</span> <span style="color: #000000;">640</span> <span style="color: #660033;">-h</span> <span style="color: #000000;">480</span> <span style="color: #660033;">-t</span> jpg <span style="color: #000000; font-weight: bold;">/</span>Users<span style="color: #000000; font-weight: bold;">/</span>USERNAME<span style="color: #000000; font-weight: bold;">/</span>Documents<span style="color: #000000; font-weight: bold;">/</span>pics<span style="color: #000000; font-weight: bold;">/</span><span style="color: #007800;">$picture</span>;<br />
<br />
<span style="color: #666666; font-style: italic;"># Take screen capture</span><br />
<span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>sbin<span style="color: #000000; font-weight: bold;">/</span>screencapture <span style="color: #660033;">-C</span> <span style="color: #660033;">-m</span> <span style="color: #660033;">-t</span> jpg <span style="color: #660033;">-x</span> <span style="color: #000000; font-weight: bold;">/</span>Users<span style="color: #000000; font-weight: bold;">/</span>USERNAME<span style="color: #000000; font-weight: bold;">/</span>Documents<span style="color: #000000; font-weight: bold;">/</span>pics<span style="color: #000000; font-weight: bold;">/</span><span style="color: #007800;">$screen</span>;<br />
<br />
<span style="color: #666666; font-style: italic;">#resize the pictures so they're not too big</span><br />
<span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>bin<span style="color: #000000; font-weight: bold;">/</span>sips <span style="color: #660033;">--resampleHeightWidthMax</span> <span style="color: #000000;">800</span> <span style="color: #000000; font-weight: bold;">/</span>Users<span style="color: #000000; font-weight: bold;">/</span>USERNAME<span style="color: #000000; font-weight: bold;">/</span>Documents<span style="color: #000000; font-weight: bold;">/</span>pics<span style="color: #000000; font-weight: bold;">/*</span>.jpg<br />
<br />
<span style="color: #666666; font-style: italic;"># Generate semaphore for connectivity by pinging NASA</span><br />
<span style="color: #007800;">isconnected</span>=$<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">ping</span> <span style="color: #660033;">-c</span> <span style="color: #000000;">1</span> www.nasa.gov <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #000000;">64</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">wc</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">awk</span> <span style="color: #ff0000;">'{print $1}'</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>;<br />
<br />
<span style="color: #666666; font-style: italic;"># Wait a few seconds to see if it's connected.</span><br />
<span style="color: #c20cb9; font-weight: bold;">sleep</span> <span style="color: #000000;">3</span>;<br />
<br />
<span style="color: #666666; font-style: italic;"># If connected...</span><br />
<span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$isconnected</span>&quot;</span> <span style="color: #660033;">-eq</span> <span style="color: #ff0000;">&quot;1&quot;</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span>; <span style="color: #000000; font-weight: bold;">then</span> <br />
<br />
&nbsp; &nbsp; <span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>bin<span style="color: #000000; font-weight: bold;">/</span><span style="color: #c20cb9; font-weight: bold;">scp</span> <span style="color: #000000; font-weight: bold;">/</span>Users<span style="color: #000000; font-weight: bold;">/</span>USERNAME<span style="color: #000000; font-weight: bold;">/</span>Documents<span style="color: #000000; font-weight: bold;">/</span>pics<span style="color: #000000; font-weight: bold;">/*</span> USERNAME<span style="color: #000000; font-weight: bold;">@</span>YOUR.SERVER.COM:<span style="color: #ff0000;">&quot;REMOTE/PATH&quot;</span>;<br />
<br />
&nbsp; &nbsp; <span style="color: #666666; font-style: italic;"># Wait a few seconds, just to make sure everything is clear and complete</span><br />
&nbsp; &nbsp; <span style="color: #c20cb9; font-weight: bold;">sleep</span> <span style="color: #000000;">4</span>;<br />
<br />
&nbsp; &nbsp; <span style="color: #666666; font-style: italic;"># delete local images</span><br />
&nbsp; &nbsp; <span style="color: #c20cb9; font-weight: bold;">rm</span> <span style="color: #660033;">-f</span> <span style="color: #000000; font-weight: bold;">/</span>Users<span style="color: #000000; font-weight: bold;">/</span>USERNAME<span style="color: #000000; font-weight: bold;">/</span>Documents<span style="color: #000000; font-weight: bold;">/</span>pics<span style="color: #000000; font-weight: bold;">/*</span>.jpg;<br />
<br />
<span style="color: #000000; font-weight: bold;">else</span> <br />
&nbsp; &nbsp; <span style="color: #666666; font-style: italic;"># If not connected.. leave captures there until next time. </span><br />
&nbsp; &nbsp; <span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;No Connection, Image not transferred or deleted.&quot;</span>;<br />
<span style="color: #000000; font-weight: bold;">fi</span></div></div>
]]></content:encoded>
			<wfw:commentRss>http://lab.joelgillman.com/archives/130_isightcapture-now-with-more-ssh/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

