You might not even like rsync. Yeah it's old. Yeah it's slow. But if you're working with Linux you're going to need to know it. In this video I walk through my favorite everyday flags for rsync. Su...

You might not even like rsync. Yeah it’s old. Yeah it’s slow. But if you’re working with Linux you’re going to need to know it.

In this video I walk through my favorite everyday flags for rsync.

Support the channel:
https://patreon.com/VeronicaExplains
https://ko-fi.com/VeronicaExplains
https://thestopbits.bandcamp.com/

Here’s a companion blog post, where I cover a bit more detail: https://vkc.sh/everyday-rsync

Also, @BreadOnPenguins made an awesome rsync video and you should check it out: https://www.youtube.com/watch?v=eifQI5uD6VQ

Lastly, I left out all of the ssh setup stuff because I made a video about that and the blog post goes into a smidge more detail. If you want to see a video covering the basics of using SSH, I made one a few years ago and it’s still pretty good: https://www.youtube.com/watch?v=3FKsdbjzBcc

Chapters:
1:18 Invoking rsync
4:05 The --delete flag for rsync
5:30 Compression flag: -z
6:02 Using tmux and rsync together
6:30 but Veronica… why not use (insert shiny object here)

  • ryper@lemmy.caEnglish
    6·
    7 hours ago

    I was planning to use rsync to ship several TB of stuff from my old NAS to my new one soon. Since we’re already talking about rsync, I guess I may as well ask if this is right way to go?

    • Suburbanl3g3nd@lemmings.worldEnglish
      7·
      5 hours ago

      I couldn’t tell you if it’s the right way but I used it on my Rpi4 to sync 4tb of stuff from my Plex drive to a backup and set a script up to have it check/mirror daily. Took a day and a half to copy and now it syncs in minutes tops when there’s new data

    • GreenKnight23@lemmy.worldEnglish
      3·
      5 hours ago

      yes, it’s the right way to go.

      rsync over ssh is the best, and works as long as rsync is installed on both systems.

      • qjkxbmwvz@startrek.websiteEnglish
        2·
        2 hours ago

        On low end CPUs you can max out the CPU before maxing out network—if you want to get fancy, you can use rsync over an unencrypted remote shell like rsh, but I would only do this if the computers were directly connected to each other by one Ethernet cable.

  • surph_ninja@lemmy.worldEnglish
    6·
    8 hours ago

    Use borg/borgmatic for your backups. Use rsync to send your differentials to your secondary & offsite backup storage.

  • NuXCOM_90Percent@lemmy.zipEnglish
    32·
    12 hours ago

    I would generally argue that rsync is not a backup solution. But it is one of the best transfer/archiving solutions.

    Yes, it is INCREDIBLY powerful and is often 90% of what people actually want/need. But to be an actual backup solution you still need infrastructure around that. Bare minimum is a crontab. But if you are actually backing something up (not just copying it to a local directory) then you need some logging/retry logic on top of that.

    At which point you are building your own borg, as it were. Which, to be clear, is a great thing to do. But… backups are incredibly important and it is very much important to understand what a backup actually needs to be.

    • tal@olio.cafeEnglish
      14·
      12 hours ago

      I would generally argue that rsync is not a backup solution.

      Yeah, if you want to use rsync specifically for backups, you’re probably better-off using something like rdiff-backup, which makes use of rsync to generate backups and store them efficiently, and drive it from something like backupninja, which will run the task periodically and notify you if it fails.

      rsync: one-way synchronization

      unison: bidirectional synchronization

      git: synchronization of text files with good interactive merging.

      rdiff-backup: rsync-based backups. I used to use this and moved to restic, as the backupninja target for rdiff-backup has kind of fallen into disrepair.

      That doesn’t mean “don’t use rsync”. I mean, rsync’s a fine tool. It’s just…not really a backup program on its own.

      • melfie@lemy.lolEnglish
        2·
        8 hours ago

        Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.

    • non_burglar@lemmy.worldEnglish
      11·
      10 hours ago

      I use rsync and a pruning script in crontab on my NFS mounts. I’ve tested it numerous times breaking containers and restoring them from backup. It works great for me at home because I don’t need anything older than 4 monthly, 4 weekly, and 7 daily backups.

      However, in my job I prefer something like bacula. The extra features and granularity of restore options makes a world of difference when someone calls because they deleted prod files.

  • mesa@piefed.socialOPEnglish
    34·
    13 hours ago

    Ive personally used rsync for backups for about…15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.

    • Eager Eagle@lemmy.worldEnglish
      5·
      7 hours ago

      It works fine if all you need is transfer, my issue with it it’s just not efficient. If you want a “time travel” feature, your only option is to duplicate data. Differential backups, compression, and encryption for off-site ones is where other tools shine.

      • bandwidthcrisis@lemmy.worldEnglish
        1·
        26 minutes ago

        I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.

        It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.

        No file deltas or compression, but it does mean that you can access the backups directly.

      • NekuSoul@lemmy.nekusoul.deEnglish
        1·
        7 hours ago

        Agree. It’s neat for file transfers and simple one-shot backups, but if you’re looking for a proper backup solution then other tools/services have advanced virtually every aspect of backups so much it pretty much always makes sense to use one of those instead.

    • confusedpuppy@lemmy.dbzer0.comEnglish
      3·
      12 hours ago

      I use rsync for many of the reasons covered in the video. It’s widely available and has a long history. To me that feels important because it’s had time to become stable and reliable. Using Linux is a hobby for me so my needs are quite low. It’s nice to have a tool that just works.

      I use it for all my backups and moving my backups to off network locations as well as file/folder transfers on my own network.

      I even made my own tool (https://codeberg.org/taters/rTransfer) to simplify all my rsync commands into readable files because rsync commands can get quite long and overwhelming. It’s especially useful chaining multiple rsync commands together to run under a single command.

      I’ve tried other backup and syncing programs and I’ve had bad experiences with all of them. Other backup programs have failed to restore my system. Syncing programs constantly stop working and I got tired of always troubleshooting. Rsync when set up properly has given me a lot less headaches.

  • ominous ocelot@leminal.spaceEnglish
    11·
    11 hours ago

    rsnapshot is a script for the purpose of repeatedly creating deduplicated copies (hardlinks) for one or more directories. You can chose how many hourly, daily, weekly,… copies you’d like to keep and it removes outdated copies automatically. It wraps rsync and ssh (public key auth) which need to be configured before.

  • i_stole_ur_taco@lemmy.caEnglish
    9·
    12 hours ago

    The thing I hate most about rsync is that I always fumble to get the right syntax and flags.

    This is a problem because once it’s working I never have to touch it ever again because it just works and keeping working. There’s not enough time to memorize the usage.

    • mesa@piefed.socialOPEnglish
      5·
      12 hours ago

      I feel this too. I have a couple of “spells” that work wonders in a literal small notebook with other one liners over the years. Its my spell book lol.

    • oddlyqueer@lemmy.mlEnglish
      2·
      12 hours ago

      This is why I still don’t know sed and awk syntax lol. I eventually get the data in the shape I need and then move on, and never imprint how they actually work. Still feel like a script kiddie every time I use them (so once every few years).

      • tal@olio.cafeEnglish
        1·
        11 hours ago

        sed can do a bunch of things, but I overwhelmingly use it for a single operation in a pipeline: the s// operation. I think that that’s worth knowing.

        sed 's/foo/bar/'

        will replace all the first text in each line matching the regex “foo” with “bar”.

        That’ll already handle a lot of cases, but a few other helpful sub-uses:

        sed 's/foo/bar/g'

        will replace all text matching regex “foo” with “bar”, even if there are more than one per line

        sed 's/\([0-9a-f]*\)/0x\1/g

        will take the text inside the backslash-escaped parens and put that matched text back in the replacement text, where one has ‘\1’. In the above example, that’s finding all hexadecimal strings and prefixing them with ‘0x’

        If you want to match a literal “/”, the easiest way to do it is to just use a different separator; if you use something other than a “/” as separator after the “s”, sed will expect that later in the expression too, like this:

        sed 's%/%SLASH%g

        will replace all instances of a “/” in the text with “SLASH”.

    • NuXCOM_90Percent@lemmy.zipEnglish
      1·
      12 hours ago

      One trick that one of my students taught me a decade or so ago is to actually make an alias to list the useful flags.

      Yes, a lot of us think we are smart and set up aliases/functions and have a huge list of them that we never remember or, even worse, ONLY remember. What I noticed her doing was having something like goodman-rsync that would just echo out a list of the most useful flags and what they actually do.

      So nine times out of 10 I just want rsync -azvh --progress ${SRC} ${DEST} but when I am doing something funky and am thinking “I vaguely recall how to do this”? dumbman rsync and I get a quick cheat sheet of what flags I have found REALLY useful in the past or even just explaining what azvh actually does without grepping past all the crap I don’t care about in the man page. And I just keep that in the repo of dotfiles I copy to machines I work on regularly.

      • muix@lemmy.sdf.orgEnglish
        3·
        9 hours ago

        tldr and atuin have been my main way of remembering complex but frequent flag combinations

        • NuXCOM_90Percent@lemmy.zipEnglish
          1·
          9 hours ago

          Yeah. There are a few useful websites I end up at that serve similar purposes.

          My usual workflow is that I need to be able to work in an airgapped environment where it is a lot easier to get “my dotfiles” approved than to ask for utility packages like that. Especially since there will inevitably be some jackass who says “You don’t know how to work without google? What are we paying you for?” because they mostly do the same task every day of their life.

          And I do find that writing the cheat sheet myself goes a long way towards me actually learning them so I don’t always need it. But I know that is very much how my brain works (I write probably hundreds of pages of notes a year… I look at maybe two pages a year).

      • tal@olio.cafeEnglish
        1·
        11 hours ago

        Most Unix commands will show a short list of the most-helpful flags if you use --help or -h.

  • tal@olio.cafeEnglish
    5·
    11 hours ago

    slow

    rsync is pretty fast, frankly. Once it’s run once, if you have -a or -t passed, it’ll synchronize mtimes. If the modification time and filesize matches, by default, rsync won’t look at a file further, so subsequent runs will be pretty fast. You can’t really beat that for speed unless you have some sort of monitoring system in place (like, filesystem-level support for identifying modifications).

  • solrize@lemmy.mlEnglish
    8·
    13 hours ago

    I’ve been using borg because of the backend encryption and because the deduplication and snapshot features are really nice. It could be interesting to have cross-archive deduplication but maybe I can get something like that by reorganizing my backups. I do use rsync for mirroring and organizing downloads, but not really for backups. It’s a synchronization program as the name implies, not really intended for backups.

  • Landless2029@lemmy.worldEnglish
    5·
    12 hours ago

    I need a breakdown like this for Rclone. I’ve got 1TB of OneDrive free and nothing to do with it.

    I’d love to setup a home server and backup some stuff to it.

  • PortNull@lemmy.dbzer0.comEnglish
    4·
    11 hours ago

    Maybe I am missing something but how does it handle snapshots?

    I use rsync all the time but only for moving data around effectively. But not for backups as it doesn’t (AFAIK) hanld snapshots

    • eleijeep@piefed.socialEnglish
      6·
      7 hours ago

      You get incremental backups (snapshots) by using

      --link-dest=DIR         hardlink to files in DIR when unchanged

      To use this you pass in the previous snapshot location as DIR and use a new destination directory for the current snapshot. This creates hard links in the new snapshot to the files which were unchanged from the previous snapshot, so only the new files are transferred, and there is no duplication of data on disk (for whole-file matches).

      This does of course require that all of the snapshots exist in the same filesystem, since you cannot hard-link across filesystems.

    • Eager Eagle@lemmy.worldEnglish
      21·
      7 hours ago

      yeah, it doesn’t, it’s just for file transfer. It’s only useful if transferring files somewhere else counts as a backup for you.

      To me, the file transfer is just a small component of a backup tool.