By: Bojidar Marinov Published: Last updated:

Transferring photos over ADB

For some odd reason, I never managed to connect my previous phone to my computer over MTP, PTP, or any of the other "usual" standards for transferring files. I suspect it has something to do with the phone's manufacturer never having tested their phone on Linux; but regardless of the case, I often needed to take photos off the phone, and removing/readding the SD card was too much pain for what it was worth, so I ended up using Draft emails to share the needed photos to my computer. Luckily, after a while I figured out a much nicer way to do that and have been using it for the last ~2.5 years now, which is what this article is about.

The method? Just throw everything through an ADB shell and catch it on the other side.

Artistic rendition of what the ADB photo transfer script looks like.

Piping tars

Android phones run Linux under the hood. Using Developer Mode (which is usually enabled by pressing the Android build number some amount of times), we can enable USB debugging and from there connect using ADB to a familiar POSIX-compatible shell.

What does that give us? Well, it's not a full-blown SSH connection, so we can't directly use SCP, and we don't have rsync on the phone pre-installed... but we do have shell access, along with a few standard tools, like ls, cat, and tar. Some testing confirms that the ADB shell does not filter the command outputs in any way, meaning we can pass arbitrary bytes through it. All of this lets us MacGyver a one-liner which transfers all photos that are found on the phone but not on the computer:

#!/usr/bin/env bash

cd /path/to/photos/sync

adb shell "cd /storage/emulated/0/DCIM/Camera/; tar -cf - $(diff -u <(adb shell "ls /storage/emulated/0/DCIM/Camera/" | grep -E "(jpg|mp4)" | sort) <(ls ../*/ | grep -E "(jpg|mp4)" | sort) | sed -nEe '1d;s/^-(.+)/\1/p')" | tar -xvf -

Note that the script expects a folder structure looking like the following:

All newly-synced photos go into sync/. From there one can manually move them to whichever folder they want, as long as that folder is nested exactly once within the parent directory. The parent directory should probably contain only photos, to not slow down the script with unrelated files.

Wait what

For a quick explanation, what this script does is a multi-step process:

  1. First, the script lists all the files in the /storage/emulated/0/DCIM/Camera/ folder of the device, using adb shell "ls ...". adb shell runs the command on the device, and is the main piece of magic that makes this work.
  2. After that, the script also lists the files that have already been synced using ls ../*/. This gives us the contents of the sync/ folder as well as all the adjacent folders.
  3. Next, the script passes both of those lists through | grep -E "(jpg|mp4)" | sort, to filter out extra files and to ensure the lists are ordered the same way. We use grep here instead of filtering with a wildcard glob (*.{jpg.mp4}), as we want ls to be listing file names and not full paths.
  4. Then, the script runs a diff on those two lists, using process substitution diff -u <(...) <(...).
  5. Next, the script passes the diff's output through a Sed command that filters out only the lines that are missing on the computer and also removes the - signs added by diff, with sed -nEe '1d;s/^-(.+)/\1/p'.
  6. Now that we have thus obtained the list of files we need to transfer, we splat it directly into a tar -cf - command. The -c flags says that we are creating a new tar archive, and the -f - flag is there to specify that we are outputting directly to stdout. The idea of using tar here is that tar allows us to pack all of the files in one "stream" of bytes, so we can transfer all of them at once.
  7. Next up, we pass that command line to adb shell, making sure to first cd to the correct directory on the phone. This cd allows us to directly extract files from the tar stream without worrying that they might have absolute path, and also lets the tar process running on the phone to find all the files we just requested.
  8. And finally, we pass everything to a tar -xvf - command. Here, the -x flag instructs tar to extract the files, -v is for printing out the file names it extracts (which lets one monitor what the script does), and -f - is again used to tell it to use the piped stream from ADB.

Evaluation

Testing this on a newer phone that does support MTP, I get almost equivalent throughput for the transfer, around 32 MiB/s. Not sure what exactly is limiting it from going faster, though; USB 2.0 should be able to go up to 480 MB/s, so it's somehow limited by the phone (adb shell "head -c 100000000 /dev/zero" | pv -s 100000000 > /dev/null also results in a measurement of ~32MiB/s).

However, when it comes to listing files... oh boy, is MTP slow. My DCIM/Camera currently rocks around 7k files. ADB takes ~2-3 seconds to list those. MTP takes over 90 seconds. (PTP seems to be transferring all files at the start, and likewise takes over a minute. ADB takes ~4 seconds to list the 12k files in storage/emulated.)

As for reliability, the only unreliable part has been USB debugging randomly turning itself off or revoking the computer's ADB authorization every now and then. Everything else has worked flawlessly: connect cable, run script, wait a few minutes, enjoy life.

I've also tried passing -z to tar to get it to compress the stream and have less data transferred, but this ends up only wasting CPU time and slowing down the transfer by about 50%.

This method of transferring files from an Android device is surprisingly useful. I've used similar commands to transfer random PDF files and voice recordings too.

My script dumps all photos in one folder, instead of sorting them by year. It's probably possible to extend it to detect the year from modification times or image metadata, but doing so would complicate it quite a bit, and I do prefer it to be simple2.

Also, my script probably breaks with file names that contain spaces. My camera has yet to spit one of those, but it will be fun when it happens. Probably something like adding s/ /\\ /g to the Sed command could do it, or perhaps changing the \1 to "\1" in that same command (but then fun stuff can happen if filenames contain $ or `). Or maybe we can use xargs on the phone instead of all that command line splatting.

adbfs-rootless manages to integrate ADB with FUSE to allow one to directly mount the needed folders as a device. Unfortunately, when I tested it out, it was much slower than my hand-rolled adb shell solution, so I stuck to what works. I do love the idea of integrating everything under the sun using FUSE filesystems, however, so I'll probably try to figure out why it's slow some day.

There is also Better ADB Sync, which does pretty much what my script does, except fancier. If you are planning to do something similar, would recommend starting with that.

In addition, there is an article by David Sebek about using rsync+ADB, which I found while writing this article. It is complicated a process to install rsync on a phone, but it can be automated, and it will likely beat adb shell-based options in the longer run.

Finally, there is adb pull, which can be used to download files from the phone, without going through the shell. I doubt it would be faster than tarring the files and passing them through a pipe, but it might be worth experimenting with some day.

Either way, this was an experimental post, much shorter than my previous ones, yet still longer than a Mastodon post. Let me know how you enjoy it!
And if you've used a similar method for transferring photos from Android (or well, files in general), I would love to hear more about it!