{
  "version": "https://jsonfeed.org/version/1",
  "title": "Ian's Digital Garden",
  "home_page_url": "https://ianwwagner.com/",
  "feed_url": "https://ianwwagner.com//tag-terminal.json",
  "description": "",
  "items": [
    {
      "id": "https://ianwwagner.com//copying-and-unarchiving-from-a-server-without-a-temp-file.html",
      "url": "https://ianwwagner.com//copying-and-unarchiving-from-a-server-without-a-temp-file.html",
      "title": "Copying and Unarchiving From a Server Without a Temp File",
      "content_html": "<p>Sometimes I want to copy files from a remote machine--usually a server I control.\nEasy; just use <code>scp</code>, right?</p>\n<p>Well, today I had a subtly different twist to the usual problem.\nI needed to transfer a ~100GB tarball to my local machine,\nand I really wanted to unarchive it so that I could get at the internal data directly.\nAnd I wanted to do it in one step, since I didn't have 200GB of free space.</p>\n<p>I happened to remember that this <em>should</em> be possible with pipes or something.\nThe tarball format is helpfully designed to allow streaming.\nBut it took me a bit to come up with the right set of commands to do this.</p>\n<p><code>scp</code> is really designed for dumping to files first.\nI found a few suggestions on StackOverflow that looked like they might work,\nbut didn't for me (might have been my shell? I use <code>fish</code> rather than <code>bash</code>).\nBut I noticed that almost all of the answers recommended using <code>ssh</code> instead,\nsince it's a bit more suited to the purpose.</p>\n<p>The basic idea is to dump the file to standard out on the remote host,\nthen pipe the ssh output into <code>tar</code> locally.\nThe <code>tar</code> flags are probably familiar or easily understandable: <code>-xvf -</code> in my case.\nThis puts <code>tar</code> into extract mode,\nenables verbose logging (so you see its progress),\nand tells it to read from stdin (<code>-</code>).\nMy <em>tarball</em> was not compressed.\nIf yours is, add the appropriate decompression flags.</p>\n<p>The SSH flags were a bit trickier.\nI discovered the <code>-C</code> flag, which enables gzip compression.\nI happen to know this dataset compresses well with gzip,\nand further that the network link between me and the remote is not the best,\nso I enabled it.\nDon't use this if your data does not compress well,\nor if it is already compressed.</p>\n<p>Another flag, <code>-e none</code>,\nI found via <a href=\"https://www.unix.com/unix-for-dummies-questions-and-answers/253941-scp-uncompress-file.html\">this unix.com forum post</a>.\nThis seemed like a good thing to enable after some research,\nsince sequences like <code>~.</code> will not be interpreted as &quot;kill the session.&quot;\nIt also prevents more subtle bugs which would look like data corruption.</p>\n<p><code>-T</code> was suggested after I pressed ChatGPT o1-preview for other flags that might be helpful.\nIt just doesn't allocate a pseudo-terminal.\nWhich we didn't need anyways.\n(Aside: ChatGPT 4o will give you some hot garbage suggestions; o1-preview was only helpful in suggesting refinements.)</p>\n<p>Finally, the command executes <code>cat</code> on the remote host to dump the tarball to <code>stdout</code>.\nI saw suggestions as well to use <code>dd</code> since you can set the block size explicitly.\nThat might improve perf in some situations if you know your hardware well.\nOr it might just be a useless attempt at premature optimization ;)</p>\n<p>Here's the final command:</p>\n<pre><code class=\"language-shell\">ssh -C -e none -T host.example.com 'cat /path/to/archive.tar' | tar -xvf -\n</code></pre>\n",
      "summary": "",
      "date_published": "2024-10-28T00:00:00-00:00",
      "image": "",
      "authors": [
        {
          "name": "Ian Wagner",
          "url": "https://fosstodon.org/@ianthetechie",
          "avatar": "media/avi.jpeg"
        }
      ],
      "tags": [
        "ssh",
        "terminal",
        "shell",
        "tar"
      ],
      "language": "en"
    }
  ]
}